2024-01-21T02:59:47.467 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-21T02:59:47.467 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652 --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652/orig.config.yaml 2024-01-21T02:59:47.503 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T02:59:47.572 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652 branch: wip-yuri8-testing-2024-01-18-0823 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7523652' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri8-testing-2024-01-18-0823 ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 0edf41a622739843d6b978b179ff3227b476dd9d valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: debuginfo: true flavor: default sha1: 0edf41a622739843d6b978b179ff3227b476dd9d thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2024-01-18-0823 sha1: 0edf41a622739843d6b978b179ff3227b476dd9d owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9599 sha1: 0edf41a622739843d6b978b179ff3227b476dd9d sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri8-testing-2024-01-18-0823 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0edf41a622739843d6b978b179ff3227b476dd9d targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCysGnDiMG3hGIFgiXaTH/oCK9ndRzJ55SkMZMS2Q83b6pe6HQhs8gb6USSF4yFdUyIfEBXiIdb/fyyj27NdoiU= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCnBrRXFdjEgSX7q1KhDE3anvh06bdhYQXczN5WiULNIIELABIokQnA9prKtCgCRS7uekcklFQ8pNArUFnIQJHc= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-19_16:13:55 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-21T02:59:47.573 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa; will attempt to use it 2024-01-21T02:59:47.573 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks 2024-01-21T02:59:47.573 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-21T02:59:47.575 INFO:teuthology.task.internal:Checking packages... 2024-01-21T02:59:47.597 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0edf41a622739843d6b978b179ff3227b476dd9d' 2024-01-21T02:59:47.597 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-21T02:59:47.597 INFO:teuthology.packaging:ref: None 2024-01-21T02:59:47.597 INFO:teuthology.packaging:tag: None 2024-01-21T02:59:47.597 INFO:teuthology.packaging:branch: wip-yuri8-testing-2024-01-18-0823 2024-01-21T02:59:47.597 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T02:59:47.597 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri8-testing-2024-01-18-0823 2024-01-21T02:59:47.738 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-766.g0edf41a6 2024-01-21T02:59:47.740 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-21T02:59:47.746 INFO:teuthology.task.internal:no buildpackages task found 2024-01-21T02:59:47.746 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-21T02:59:47.768 INFO:teuthology.task.internal:Saving configuration 2024-01-21T02:59:47.782 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-21T02:59:47.790 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T02:59:47.815 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 02:54:48.390935', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCysGnDiMG3hGIFgiXaTH/oCK9ndRzJ55SkMZMS2Q83b6pe6HQhs8gb6USSF4yFdUyIfEBXiIdb/fyyj27NdoiU='} 2024-01-21T02:59:47.842 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 02:54:48.392521', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCnBrRXFdjEgSX7q1KhDE3anvh06bdhYQXczN5WiULNIIELABIokQnA9prKtCgCRS7uekcklFQ8pNArUFnIQJHc='} 2024-01-21T02:59:47.842 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-21T02:59:47.848 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-01-21T02:59:47.849 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-01-21T02:59:47.849 INFO:teuthology.run_tasks:Running task console_log... 2024-01-21T02:59:47.919 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0ad0aaaee0>, signals=[15]) 2024-01-21T02:59:47.919 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-21T02:59:47.928 INFO:teuthology.task.internal:Opening connections... 2024-01-21T02:59:47.928 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-01-21T02:59:47.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T02:59:48.036 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-01-21T02:59:48.037 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T02:59:48.124 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-21T02:59:48.132 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-01-21T02:59:48.150 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-01-21T02:59:48.150 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-01-21T02:59:48.208 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2024-01-21T02:59:48.208 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="9" 2024-01-21T02:59:48.208 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2024-01-21T02:59:48.208 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2024-01-21T02:59:48.208 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="9" 2024-01-21T02:59:48.208 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el9" 2024-01-21T02:59:48.208 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-21T02:59:48.208 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2024-01-21T02:59:48.208 INFO:teuthology.orchestra.run.smithi007.stdout:LOGO="fedora-logo-icon" 2024-01-21T02:59:48.209 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-21T02:59:48.209 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2024-01-21T02:59:48.209 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T02:59:48.209 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-21T02:59:48.209 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T02:59:48.209 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-01-21T02:59:48.233 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-01-21T02:59:48.250 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-01-21T02:59:48.250 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-01-21T02:59:48.307 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-01-21T02:59:48.307 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-01-21T02:59:48.307 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-01-21T02:59:48.307 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-01-21T02:59:48.307 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-01-21T02:59:48.307 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-01-21T02:59:48.307 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-21T02:59:48.307 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-01-21T02:59:48.307 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-01-21T02:59:48.307 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-21T02:59:48.307 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-01-21T02:59:48.308 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T02:59:48.308 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-21T02:59:48.308 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T02:59:48.308 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-01-21T02:59:48.325 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-21T02:59:48.334 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-21T02:59:48.343 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-21T02:59:48.343 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-01-21T02:59:48.345 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-01-21T02:59:48.363 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-21T02:59:48.370 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-21T02:59:48.370 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-01-21T02:59:48.402 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-01-21T02:59:48.491 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-21T02:59:48.538 INFO:teuthology.run_tasks:Running task kernel... 2024-01-21T02:59:48.552 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-21T02:59:48.553 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-21T02:59:48.553 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T02:59:48.553 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T02:59:48.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T02:59:48.570 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-01-21T02:59:48.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T02:59:48.571 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-01-21T02:59:48.627 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-361.el9.x86_64 2024-01-21T02:59:48.628 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.14.0-361.el9.x86_64 2024-01-21T02:59:48.628 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum install -y kernel 2024-01-21T02:59:48.629 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-361.el9.x86_64 2024-01-21T02:59:48.629 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.14.0-361.el9.x86_64 2024-01-21T02:59:48.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2024-01-21T02:59:50.898 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 9.4 MB/s | 6.4 MB 00:00 2024-01-21T02:59:51.081 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 8.0 MB/s | 6.4 MB 00:00 2024-01-21T02:59:52.532 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-01-21T02:59:52.772 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-01-21T02:59:54.480 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-01-21T02:59:55.381 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.0 MB 00:02 2024-01-21T02:59:57.890 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-01-21T03:00:02.782 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 5.9 MB 00:01 2024-01-21T03:00:04.399 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-01-21T03:00:05.513 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 1.9 MB/s | 19 MB 00:09 2024-01-21T03:00:06.501 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 10 MB/s | 20 MB 00:01 2024-01-21T03:00:10.572 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 5.9 MB 00:01 2024-01-21T03:00:11.300 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-01-21T03:00:12.098 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 15 kB 00:00 2024-01-21T03:00:13.474 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-21T03:00:13.474 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-21T03:00:13.563 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:00:13.574 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:00:13.575 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-01-21T03:00:13.575 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:00:13.575 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:00:13.575 INFO:teuthology.orchestra.run.smithi177.stdout: kernel x86_64 5.14.0-409.el9 CentOS-BaseOS 6.0 M 2024-01-21T03:00:13.575 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-01-21T03:00:13.575 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core x86_64 5.14.0-409.el9 CentOS-BaseOS 21 M 2024-01-21T03:00:13.575 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules x86_64 5.14.0-409.el9 CentOS-BaseOS 39 M 2024-01-21T03:00:13.575 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core x86_64 5.14.0-409.el9 CentOS-BaseOS 33 M 2024-01-21T03:00:13.576 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:00:13.576 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:00:13.576 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:00:13.576 INFO:teuthology.orchestra.run.smithi177.stdout:Install 4 Packages 2024-01-21T03:00:13.576 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:00:13.577 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 99 M 2024-01-21T03:00:13.577 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 122 M 2024-01-21T03:00:13.577 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:00:14.002 INFO:teuthology.orchestra.run.smithi177.stdout:(1/4): kernel-5.14.0-409.el9.x86_64.rpm 14 MB/s | 6.0 MB 00:00 2024-01-21T03:00:14.585 INFO:teuthology.orchestra.run.smithi177.stdout:(2/4): kernel-core-5.14.0-409.el9.x86_64.rpm 21 MB/s | 21 MB 00:01 2024-01-21T03:00:14.632 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 20 MB 00:02 2024-01-21T03:00:15.527 INFO:teuthology.orchestra.run.smithi177.stdout:(3/4): kernel-modules-5.14.0-409.el9.x86_64.rpm 20 MB/s | 39 MB 00:01 2024-01-21T03:00:16.035 INFO:teuthology.orchestra.run.smithi177.stdout:(4/4): kernel-modules-core-5.14.0-409.el9.x86_6 16 MB/s | 33 MB 00:02 2024-01-21T03:00:16.038 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:00:16.039 INFO:teuthology.orchestra.run.smithi177.stdout:Total 40 MB/s | 99 MB 00:02 2024-01-21T03:00:16.039 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:00:16.401 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:00:16.401 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:00:17.436 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:00:17.436 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:00:19.590 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-01-21T03:00:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:00:20.726 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-core-5.14.0-409.el9.x86_64 1/4 2024-01-21T03:00:20.861 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-21T03:00:21.806 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-21T03:00:21.806 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-21T03:00:21.901 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:00:21.917 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:00:21.917 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-01-21T03:00:21.917 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:00:21.917 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:00:21.917 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-409.el9 CentOS-BaseOS 6.0 M 2024-01-21T03:00:21.917 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-01-21T03:00:21.917 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core x86_64 5.14.0-409.el9 CentOS-BaseOS 21 M 2024-01-21T03:00:21.917 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules x86_64 5.14.0-409.el9 CentOS-BaseOS 39 M 2024-01-21T03:00:21.917 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core x86_64 5.14.0-409.el9 CentOS-BaseOS 33 M 2024-01-21T03:00:21.917 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:00:21.917 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:00:21.917 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:00:21.917 INFO:teuthology.orchestra.run.smithi007.stdout:Install 4 Packages 2024-01-21T03:00:21.918 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:00:21.918 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 99 M 2024-01-21T03:00:21.918 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 122 M 2024-01-21T03:00:21.918 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:00:22.333 INFO:teuthology.orchestra.run.smithi007.stdout:(1/4): kernel-5.14.0-409.el9.x86_64.rpm 14 MB/s | 6.0 MB 00:00 2024-01-21T03:00:23.025 INFO:teuthology.orchestra.run.smithi007.stdout:(2/4): kernel-core-5.14.0-409.el9.x86_64.rpm 19 MB/s | 21 MB 00:01 2024-01-21T03:00:23.578 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-21T03:00:23.794 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-21T03:00:23.967 INFO:teuthology.orchestra.run.smithi007.stdout:(3/4): kernel-modules-core-5.14.0-409.el9.x86_6 20 MB/s | 33 MB 00:01 2024-01-21T03:00:24.451 INFO:teuthology.orchestra.run.smithi007.stdout:(4/4): kernel-modules-5.14.0-409.el9.x86_64.rpm 15 MB/s | 39 MB 00:02 2024-01-21T03:00:24.454 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:00:24.454 INFO:teuthology.orchestra.run.smithi007.stdout:Total 39 MB/s | 99 MB 00:02 2024-01-21T03:00:24.454 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:00:24.877 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:00:24.877 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:00:25.972 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:00:25.972 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:00:28.463 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-21T03:00:28.492 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:00:28.576 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-5.14.0-409.el9.x86_64 4/4 2024-01-21T03:00:29.194 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-core-5.14.0-409.el9.x86_64 1/4 2024-01-21T03:00:29.329 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-21T03:00:32.203 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-21T03:00:32.391 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-21T03:00:33.276 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-21T03:00:37.439 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-21T03:00:37.545 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-5.14.0-409.el9.x86_64 4/4 2024-01-21T03:00:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-21T03:01:02.450 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 4/4 2024-01-21T03:01:02.450 INFO:teuthology.orchestra.run.smithi177.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-21T03:01:02.450 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:01:02.519 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 4/4 2024-01-21T03:01:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-5.14.0-409.el9.x86_64 4/4 2024-01-21T03:01:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-5.14.0-409.el9.x86_64 1/4 2024-01-21T03:01:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-21T03:01:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-21T03:01:03.349 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-21T03:01:03.349 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:01:03.349 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:01:03.349 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-5.14.0-409.el9.x86_64 2024-01-21T03:01:03.349 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core-5.14.0-409.el9.x86_64 2024-01-21T03:01:03.350 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-5.14.0-409.el9.x86_64 2024-01-21T03:01:03.350 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core-5.14.0-409.el9.x86_64 2024-01-21T03:01:03.350 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:01:03.350 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:01:03.843 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T03:01:03.979 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-21T03:01:03.979 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-21T03:01:03.979 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-409.el9.x86_64 2024-01-21T03:01:03.980 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T03:01:03.980 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T03:01:03.980 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-01-21T03:01:03.980 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T03:01:03.980 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T03:01:04.088 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-21T03:01:04.088 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-21T03:01:04.089 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -qi grub2-tools 2024-01-21T03:01:04.122 INFO:teuthology.orchestra.run.smithi177.stdout:Name : grub2-tools 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:Epoch : 1 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:Version : 2.06 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:Release : 68.el9 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: x86_64 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:Group : Unspecified 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:Size : 8264254 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:License : GPLv3+ 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:Packager : builder@centos.org 2024-01-21T03:01:04.123 INFO:teuthology.orchestra.run.smithi177.stdout:Vendor : CentOS 2024-01-21T03:01:04.124 INFO:teuthology.orchestra.run.smithi177.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T03:01:04.124 INFO:teuthology.orchestra.run.smithi177.stdout:Summary : Support tools for GRUB. 2024-01-21T03:01:04.124 INFO:teuthology.orchestra.run.smithi177.stdout:Description : 2024-01-21T03:01:04.124 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:01:04.124 INFO:teuthology.orchestra.run.smithi177.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T03:01:04.124 INFO:teuthology.orchestra.run.smithi177.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T03:01:04.124 INFO:teuthology.orchestra.run.smithi177.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T03:01:04.124 INFO:teuthology.orchestra.run.smithi177.stdout:hardware devices. 2024-01-21T03:01:04.124 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:01:04.124 INFO:teuthology.orchestra.run.smithi177.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T03:01:04.125 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T03:01:04.125 INFO:teuthology.task.kernel:Updating grub on smithi177 to boot 5.14.0-409.el9.x86_64 2024-01-21T03:01:04.125 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T03:01:04.780 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-01-21T03:01:06.209 INFO:teuthology.orchestra.run.smithi177.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-21T03:01:06.236 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-01-21T03:01:06.243 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-01-21T03:01:06.299 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.KmZ4DffK19 2024-01-21T03:01:06.299 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KmZ4DffK19 2024-01-21T03:01:06.377 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.KmZ4DffK19 2024-01-21T03:01:06.514 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.KmZ4DffK19 is 6KB 2024-01-21T03:01:06.561 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.KmZ4DffK19 2024-01-21T03:01:06.575 DEBUG:teuthology.orchestra.run.smithi177:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T03:01:06.640 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-21T03:01:06.640 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-21T03:01:06.640 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-21T03:01:06.640 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64.conf 2024-01-21T03:01:06.641 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64 2024-01-21T03:01:06.797 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-01-21T03:01:13.425 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 4/4 2024-01-21T03:01:13.426 INFO:teuthology.orchestra.run.smithi007.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-21T03:01:13.426 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:01:13.498 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 4/4 2024-01-21T03:01:13.978 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-5.14.0-409.el9.x86_64 4/4 2024-01-21T03:01:13.978 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-409.el9.x86_64 1/4 2024-01-21T03:01:13.978 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-21T03:01:13.979 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-21T03:01:14.418 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-21T03:01:14.418 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:01:14.418 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:01:14.418 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-5.14.0-409.el9.x86_64 2024-01-21T03:01:14.418 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core-5.14.0-409.el9.x86_64 2024-01-21T03:01:14.418 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-5.14.0-409.el9.x86_64 2024-01-21T03:01:14.419 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core-5.14.0-409.el9.x86_64 2024-01-21T03:01:14.419 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:01:14.419 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:01:14.956 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T03:01:15.101 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-21T03:01:15.101 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-21T03:01:15.101 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-409.el9.x86_64 2024-01-21T03:01:15.101 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T03:01:15.101 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T03:01:15.101 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-21T03:01:15.101 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T03:01:15.102 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T03:01:15.213 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-21T03:01:15.214 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-21T03:01:15.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2024-01-21T03:01:15.257 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2024-01-21T03:01:15.257 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.06 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 68.el9 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:Group : Unspecified 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 8264254 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : builder@centos.org 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : CentOS 2024-01-21T03:01:15.258 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T03:01:15.259 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2024-01-21T03:01:15.259 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2024-01-21T03:01:15.259 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:01:15.259 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T03:01:15.259 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T03:01:15.259 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T03:01:15.259 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2024-01-21T03:01:15.259 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:01:15.259 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T03:01:15.260 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T03:01:15.261 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 5.14.0-409.el9.x86_64 2024-01-21T03:01:15.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T03:01:15.817 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2024-01-21T03:01:17.187 INFO:teuthology.orchestra.run.smithi007.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-21T03:01:17.217 INFO:teuthology.orchestra.run.smithi007.stderr:done 2024-01-21T03:01:17.219 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-01-21T03:01:17.235 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.KtNLX2Pb2H 2024-01-21T03:01:17.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KtNLX2Pb2H 2024-01-21T03:01:17.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.KtNLX2Pb2H 2024-01-21T03:01:17.458 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.KtNLX2Pb2H is 6KB 2024-01-21T03:01:17.507 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.KtNLX2Pb2H 2024-01-21T03:01:17.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T03:01:17.590 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-21T03:01:17.590 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-21T03:01:17.590 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-21T03:01:17.590 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64.conf 2024-01-21T03:01:17.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64 2024-01-21T03:01:17.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2024-01-21T03:01:36.819 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-01-21T03:01:36.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-01-21T03:01:36.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T03:01:47.762 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-21T03:01:47.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-01-21T03:01:47.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T03:02:11.516 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-01-21T03:02:20.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-01-21T03:02:20.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T03:02:38.912 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-01-21T03:02:47.774 DEBUG:teuthology.orchestra.remote:timed out 2024-01-21T03:02:50.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-01-21T03:02:50.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T03:02:51.313 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-01-21T03:02:51.703 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-01-21T03:02:51.703 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-409.el9.x86_64"... 2024-01-21T03:02:51.704 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-01-21T03:02:51.717 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-409.el9.x86_64 2024-01-21T03:02:51.718 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-409.el9.x86_64 vs 5.14.0-409.el9.x86_64 2024-01-21T03:02:51.718 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T03:02:51.718 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T03:02:52.718 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-01-21T03:02:52.719 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T03:02:52.806 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-01-21T03:02:52.846 DEBUG:teuthology.parallel:result is None 2024-01-21T03:02:56.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-01-21T03:02:56.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T03:02:57.067 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-01-21T03:02:57.503 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-01-21T03:02:57.503 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-409.el9.x86_64"... 2024-01-21T03:02:57.503 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-01-21T03:02:57.516 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-409.el9.x86_64 2024-01-21T03:02:57.517 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-409.el9.x86_64 vs 5.14.0-409.el9.x86_64 2024-01-21T03:02:57.517 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T03:02:57.517 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T03:02:58.517 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-21T03:02:58.518 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T03:02:58.580 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2024-01-21T03:02:58.631 DEBUG:teuthology.parallel:result is None 2024-01-21T03:02:58.631 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-21T03:02:58.640 INFO:teuthology.task.internal:Creating test directory... 2024-01-21T03:02:58.640 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T03:02:58.643 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T03:02:58.661 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-21T03:02:58.667 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-21T03:02:58.676 INFO:teuthology.task.internal:Creating archive directory... 2024-01-21T03:02:58.676 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T03:02:58.697 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T03:02:58.733 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-21T03:02:58.741 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-21T03:02:58.741 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T03:02:58.752 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T03:02:58.788 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T03:02:58.800 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T03:02:58.801 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T03:02:58.814 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T03:02:58.816 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-21T03:02:58.823 INFO:teuthology.task.internal:Configuring sudo... 2024-01-21T03:02:58.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T03:02:58.844 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T03:02:58.886 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-21T03:02:58.897 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-21T03:02:58.897 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T03:02:58.911 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T03:02:58.942 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T03:02:59.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T03:02:59.051 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T03:02:59.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T03:02:59.173 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:02:59.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T03:02:59.241 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T03:02:59.285 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T03:02:59.334 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T03:02:59.419 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T03:02:59.460 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:02:59.460 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T03:02:59.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-01-21T03:02:59.531 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-01-21T03:02:59.577 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T03:02:59.619 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T03:02:59.949 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-21T03:02:59.958 INFO:teuthology.task.internal:Starting timer... 2024-01-21T03:02:59.958 INFO:teuthology.run_tasks:Running task pcp... 2024-01-21T03:02:59.968 INFO:teuthology.run_tasks:Running task selinux... 2024-01-21T03:02:59.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2024-01-21T03:03:00.048 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2024-01-21T03:03:00.050 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-01-21T03:03:00.116 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-01-21T03:03:00.118 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-21T03:03:00.118 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2024-01-21T03:03:00.134 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2024-01-21T03:03:00.135 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-01-21T03:03:00.153 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-01-21T03:03:00.153 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-01-21T03:03:00.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T03:03:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T03:03:00.201 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T03:03:00.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T03:03:00.229 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-21T03:03:00.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2024-01-21T03:03:00.281 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-01-21T03:03:00.314 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-21T03:03:00.323 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-21T03:03:00.328 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-21T03:03:00.449 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-21T03:03:00.463 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-21T03:03:00.465 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-21T03:07:01.230 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-01-21T03:07:01.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-01-21T03:07:01.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T03:07:01.337 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-01-21T03:07:01.414 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-01-21T03:07:01.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-01-21T03:07:01.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T03:07:01.504 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-01-21T03:07:01.582 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-01-21T03:07:01.583 INFO:teuthology.run_tasks:Running task clock... 2024-01-21T03:07:01.597 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-21T03:07:01.597 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T03:07:01.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T03:07:01.600 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T03:07:01.600 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T03:07:01.629 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T03:07:01.645 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T03:07:01.653 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T03:07:01.670 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T03:07:01.674 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2024-01-21T03:07:01.688 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2024-01-21T03:07:01.699 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-01-21T03:07:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T03:07:01.713 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-01-21T03:07:01.721 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T03:07:01.730 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T03:07:01.746 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T03:07:01.777 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-01-21T03:07:01.780 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T03:07:01.780 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-01-21T03:07:01.780 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T03:07:01.780 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T03:07:01.780 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T03:07:01.781 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T03:07:01.799 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-01-21T03:07:01.803 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T03:07:01.803 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-01-21T03:07:01.804 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T03:07:01.804 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T03:07:01.804 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T03:07:01.804 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T03:07:01.804 INFO:teuthology.run_tasks:Running task install... 2024-01-21T03:07:01.815 DEBUG:teuthology.task.install:project ceph 2024-01-21T03:07:01.816 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'}} 2024-01-21T03:07:01.816 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'} 2024-01-21T03:07:01.816 INFO:teuthology.task.install:Using flavor: default 2024-01-21T03:07:01.822 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-01-21T03:07:01.822 INFO:teuthology.task.install:extra packages: [] 2024-01-21T03:07:01.822 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'tag': None, 'wait_for_package': False} 2024-01-21T03:07:01.823 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T03:07:01.825 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'tag': None, 'wait_for_package': False} 2024-01-21T03:07:01.825 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T03:07:01.982 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/ 2024-01-21T03:07:01.982 INFO:teuthology.task.install.rpm:Package version is 19.0.0-766.g0edf41a6 2024-01-21T03:07:02.043 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/ 2024-01-21T03:07:02.044 INFO:teuthology.task.install.rpm:Package version is 19.0.0-766.g0edf41a6 2024-01-21T03:07:02.162 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T03:07:02.162 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:07:02.162 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T03:07:02.193 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-01-21T03:07:02.193 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0edf41a622739843d6b978b179ff3227b476dd9d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T03:07:02.212 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T03:07:02.213 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:07:02.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T03:07:02.241 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-01-21T03:07:02.241 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0edf41a622739843d6b978b179ff3227b476dd9d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T03:07:02.262 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T03:07:02.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T03:07:02.343 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T03:07:02.389 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T03:07:02.410 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-01-21T03:07:02.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-01-21T03:07:02.455 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2024-01-21T03:07:02.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2024-01-21T03:07:02.660 INFO:teuthology.orchestra.run.smithi177.stdout:73 files removed 2024-01-21T03:07:02.687 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw 2024-01-21T03:07:02.712 INFO:teuthology.orchestra.run.smithi007.stdout:73 files removed 2024-01-21T03:07:02.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw 2024-01-21T03:07:03.394 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 205 kB/s | 80 kB 00:00 2024-01-21T03:07:03.396 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 239 kB/s | 80 kB 00:00 2024-01-21T03:07:03.711 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 35 kB/s | 9.5 kB 00:00 2024-01-21T03:07:03.831 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2024-01-21T03:07:04.112 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-01-21T03:07:04.257 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-01-21T03:07:04.692 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 16 MB/s | 6.4 MB 00:00 2024-01-21T03:07:04.764 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 10 MB/s | 6.4 MB 00:00 2024-01-21T03:07:06.243 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-01-21T03:07:06.339 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-01-21T03:07:08.161 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.0 MB 00:01 2024-01-21T03:07:08.963 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 3.5 MB/s | 8.0 MB 00:02 2024-01-21T03:07:12.084 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 19 MB 00:02 2024-01-21T03:07:12.847 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 19 MB 00:02 2024-01-21T03:07:16.993 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 5.9 MB 00:01 2024-01-21T03:07:17.156 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 5.9 MB 00:00 2024-01-21T03:07:18.491 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 15 kB 00:00 2024-01-21T03:07:19.335 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 15 kB 00:01 2024-01-21T03:07:21.204 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2024-01-21T03:07:21.317 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 20 MB 00:02 2024-01-21T03:07:26.150 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-01-21T03:07:26.220 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-01-21T03:07:28.667 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:07:28.680 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 8.6 M 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 3.4 M 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 3.1 M 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 5.1 M 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 19 M 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 25 k 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-21T03:07:28.681 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 742 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 499 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 4.8 M 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 45 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 130 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 161 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 321 k 2024-01-21T03:07:28.682 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 298 k 2024-01-21T03:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 99 k 2024-01-21T03:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-21T03:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-21T03:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout:Install 27 Packages 2024-01-21T03:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 2 Packages 2024-01-21T03:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 55 M 2024-01-21T03:07:28.684 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:07:28.816 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:07:28.829 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 8.6 M 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 3.4 M 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 3.1 M 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 5.1 M 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 19 M 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 25 k 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-21T03:07:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 742 k 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 499 k 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 4.8 M 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 45 k 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 130 k 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 161 k 2024-01-21T03:07:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-21T03:07:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 321 k 2024-01-21T03:07:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 298 k 2024-01-21T03:07:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 99 k 2024-01-21T03:07:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-21T03:07:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-21T03:07:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:07:28.832 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:07:28.832 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:07:28.832 INFO:teuthology.orchestra.run.smithi007.stdout:Install 27 Packages 2024-01-21T03:07:28.832 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 2 Packages 2024-01-21T03:07:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:07:28.833 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 55 M 2024-01-21T03:07:28.833 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:07:32.329 INFO:teuthology.orchestra.run.smithi007.stdout:(1/29): ceph-base-19.0.0-766.g0edf41a6.el9.x86_ 1.5 MB/s | 5.1 MB 00:03 2024-01-21T03:07:32.429 INFO:teuthology.orchestra.run.smithi007.stdout:(2/29): ceph-selinux-19.0.0-766.g0edf41a6.el9.x 252 kB/s | 25 kB 00:00 2024-01-21T03:07:32.813 INFO:teuthology.orchestra.run.smithi007.stdout:(3/29): libcephfs2-19.0.0-766.g0edf41a6.el9.x86 1.9 MB/s | 742 kB 00:00 2024-01-21T03:07:33.105 INFO:teuthology.orchestra.run.smithi007.stdout:(4/29): libradosstriper1-19.0.0-766.g0edf41a6.e 1.7 MB/s | 499 kB 00:00 2024-01-21T03:07:33.250 INFO:teuthology.orchestra.run.smithi177.stdout:(1/29): ceph-radosgw-19.0.0-766.g0edf41a6.el9.x 2.0 MB/s | 8.6 MB 00:04 2024-01-21T03:07:33.359 INFO:teuthology.orchestra.run.smithi177.stdout:(2/29): ceph-selinux-19.0.0-766.g0edf41a6.el9.x 233 kB/s | 25 kB 00:00 2024-01-21T03:07:33.676 INFO:teuthology.orchestra.run.smithi177.stdout:(3/29): ceph-base-19.0.0-766.g0edf41a6.el9.x86_ 1.1 MB/s | 5.1 MB 00:04 2024-01-21T03:07:33.709 INFO:teuthology.orchestra.run.smithi177.stdout:(4/29): libcephfs2-19.0.0-766.g0edf41a6.el9.x86 2.1 MB/s | 742 kB 00:00 2024-01-21T03:07:34.126 INFO:teuthology.orchestra.run.smithi177.stdout:(5/29): libradosstriper1-19.0.0-766.g0edf41a6.e 1.1 MB/s | 499 kB 00:00 2024-01-21T03:07:34.185 INFO:teuthology.orchestra.run.smithi177.stdout:(6/29): python3-ceph-argparse-19.0.0-766.g0edf4 776 kB/s | 45 kB 00:00 2024-01-21T03:07:34.293 INFO:teuthology.orchestra.run.smithi177.stdout:(7/29): python3-ceph-common-19.0.0-766.g0edf41a 1.2 MB/s | 130 kB 00:00 2024-01-21T03:07:34.410 INFO:teuthology.orchestra.run.smithi177.stdout:(8/29): python3-cephfs-19.0.0-766.g0edf41a6.el9 1.4 MB/s | 161 kB 00:00 2024-01-21T03:07:34.619 INFO:teuthology.orchestra.run.smithi177.stdout:(9/29): python3-rados-19.0.0-766.g0edf41a6.el9. 1.5 MB/s | 321 kB 00:00 2024-01-21T03:07:34.811 INFO:teuthology.orchestra.run.smithi177.stdout:(10/29): python3-rbd-19.0.0-766.g0edf41a6.el9.x 1.5 MB/s | 298 kB 00:00 2024-01-21T03:07:34.878 INFO:teuthology.orchestra.run.smithi177.stdout:(11/29): python3-rgw-19.0.0-766.g0edf41a6.el9.x 1.5 MB/s | 99 kB 00:00 2024-01-21T03:07:35.137 INFO:teuthology.orchestra.run.smithi177.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 410 kB/s | 106 kB 00:00 2024-01-21T03:07:35.155 INFO:teuthology.orchestra.run.smithi007.stdout:(5/29): librgw2-19.0.0-766.g0edf41a6.el9.x86_64 2.3 MB/s | 4.8 MB 00:02 2024-01-21T03:07:35.195 INFO:teuthology.orchestra.run.smithi177.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-01-21T03:07:35.229 INFO:teuthology.orchestra.run.smithi177.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-01-21T03:07:35.264 INFO:teuthology.orchestra.run.smithi007.stdout:(6/29): ceph-radosgw-19.0.0-766.g0edf41a6.el9.x 1.4 MB/s | 8.6 MB 00:06 2024-01-21T03:07:35.280 INFO:teuthology.orchestra.run.smithi007.stdout:(7/29): python3-ceph-argparse-19.0.0-766.g0edf4 360 kB/s | 45 kB 00:00 2024-01-21T03:07:35.287 INFO:teuthology.orchestra.run.smithi177.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-01-21T03:07:35.329 INFO:teuthology.orchestra.run.smithi177.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-01-21T03:07:35.363 INFO:teuthology.orchestra.run.smithi177.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-01-21T03:07:35.381 INFO:teuthology.orchestra.run.smithi007.stdout:(8/29): python3-cephfs-19.0.0-766.g0edf41a6.el9 1.6 MB/s | 161 kB 00:00 2024-01-21T03:07:35.439 INFO:teuthology.orchestra.run.smithi007.stdout:(9/29): python3-ceph-common-19.0.0-766.g0edf41a 741 kB/s | 130 kB 00:00 2024-01-21T03:07:35.505 INFO:teuthology.orchestra.run.smithi177.stdout:(18/29): librgw2-19.0.0-766.g0edf41a6.el9.x86_6 2.7 MB/s | 4.8 MB 00:01 2024-01-21T03:07:35.530 INFO:teuthology.orchestra.run.smithi177.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.8 MB/s | 309 kB 00:00 2024-01-21T03:07:35.539 INFO:teuthology.orchestra.run.smithi007.stdout:(10/29): python3-rados-19.0.0-766.g0edf41a6.el9 2.0 MB/s | 321 kB 00:00 2024-01-21T03:07:35.572 INFO:teuthology.orchestra.run.smithi177.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-01-21T03:07:35.614 INFO:teuthology.orchestra.run.smithi177.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-01-21T03:07:35.632 INFO:teuthology.orchestra.run.smithi007.stdout:(11/29): python3-rgw-19.0.0-766.g0edf41a6.el9.x 1.1 MB/s | 99 kB 00:00 2024-01-21T03:07:35.656 INFO:teuthology.orchestra.run.smithi177.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-01-21T03:07:35.698 INFO:teuthology.orchestra.run.smithi007.stdout:(12/29): python3-rbd-19.0.0-766.g0edf41a6.el9.x 1.1 MB/s | 298 kB 00:00 2024-01-21T03:07:35.731 INFO:teuthology.orchestra.run.smithi177.stdout:(23/29): libarrow-9.0.0-10.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-01-21T03:07:35.757 INFO:teuthology.orchestra.run.smithi007.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 846 kB/s | 106 kB 00:00 2024-01-21T03:07:35.765 INFO:teuthology.orchestra.run.smithi177.stdout:(24/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 838 kB 00:00 2024-01-21T03:07:35.782 INFO:teuthology.orchestra.run.smithi007.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-01-21T03:07:35.790 INFO:teuthology.orchestra.run.smithi177.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-01-21T03:07:35.816 INFO:teuthology.orchestra.run.smithi007.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 755 kB/s | 44 kB 00:00 2024-01-21T03:07:35.849 INFO:teuthology.orchestra.run.smithi177.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-01-21T03:07:35.858 INFO:teuthology.orchestra.run.smithi007.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-01-21T03:07:35.883 INFO:teuthology.orchestra.run.smithi007.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-01-21T03:07:35.909 INFO:teuthology.orchestra.run.smithi007.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 827 kB/s | 41 kB 00:00 2024-01-21T03:07:36.226 INFO:teuthology.orchestra.run.smithi007.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 904 kB/s | 309 kB 00:00 2024-01-21T03:07:36.434 INFO:teuthology.orchestra.run.smithi007.stdout:(20/29): ceph-common-19.0.0-766.g0edf41a6.el9.x 2.6 MB/s | 19 MB 00:07 2024-01-21T03:07:36.460 INFO:teuthology.orchestra.run.smithi007.stdout:(21/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 111 kB/s | 26 kB 00:00 2024-01-21T03:07:36.518 INFO:teuthology.orchestra.run.smithi007.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-01-21T03:07:36.643 INFO:teuthology.orchestra.run.smithi007.stdout:(23/29): liboath-2.6.7-2.el9.x86_64.rpm 233 kB/s | 49 kB 00:00 2024-01-21T03:07:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-01-21T03:07:36.802 INFO:teuthology.orchestra.run.smithi007.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 838 kB 00:00 2024-01-21T03:07:36.849 INFO:teuthology.orchestra.run.smithi177.stdout:(27/29): librbd1-19.0.0-766.g0edf41a6.el9.x86_6 3.1 MB/s | 3.1 MB 00:00 2024-01-21T03:07:37.194 INFO:teuthology.orchestra.run.smithi007.stdout:(26/29): libarrow-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2024-01-21T03:07:37.311 INFO:teuthology.orchestra.run.smithi007.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 2.9 MB/s | 1.6 MB 00:00 2024-01-21T03:07:37.658 INFO:teuthology.orchestra.run.smithi177.stdout:(28/29): librados2-19.0.0-766.g0edf41a6.el9.x86 1.8 MB/s | 3.4 MB 00:01 2024-01-21T03:07:38.120 INFO:teuthology.orchestra.run.smithi007.stdout:(28/29): librados2-19.0.0-766.g0edf41a6.el9.x86 2.5 MB/s | 3.4 MB 00:01 2024-01-21T03:07:38.253 INFO:teuthology.orchestra.run.smithi007.stdout:(29/29): librbd1-19.0.0-766.g0edf41a6.el9.x86_6 3.0 MB/s | 3.1 MB 00:01 2024-01-21T03:07:38.260 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:07:38.260 INFO:teuthology.orchestra.run.smithi007.stdout:Total 5.9 MB/s | 55 MB 00:09 2024-01-21T03:07:38.260 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:07:38.283 INFO:teuthology.orchestra.run.smithi177.stdout:(29/29): ceph-common-19.0.0-766.g0edf41a6.el9.x 2.1 MB/s | 19 MB 00:09 2024-01-21T03:07:38.286 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:07:38.287 INFO:teuthology.orchestra.run.smithi177.stdout:Total 5.8 MB/s | 55 MB 00:09 2024-01-21T03:07:38.287 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:07:38.300 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:07:38.300 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:07:38.322 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:07:38.322 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:07:38.776 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:07:38.777 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:07:38.788 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:07:38.788 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:07:40.156 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:07:40.200 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:07:40.241 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-21T03:07:40.287 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-21T03:07:40.287 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-01-21T03:07:40.316 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-01-21T03:07:40.338 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-01-21T03:07:40.368 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-01-21T03:07:40.711 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-01-21T03:07:40.725 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-01-21T03:07:40.768 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 5/31 2024-01-21T03:07:40.770 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 5/31 2024-01-21T03:07:40.860 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 5/31 2024-01-21T03:07:40.862 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 5/31 2024-01-21T03:07:40.948 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 6/31 2024-01-21T03:07:40.949 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 6/31 2024-01-21T03:07:40.977 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 7/31 2024-01-21T03:07:40.980 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 7/31 2024-01-21T03:07:41.034 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 7/31 2024-01-21T03:07:41.040 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 7/31 2024-01-21T03:07:41.054 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-01-21T03:07:41.069 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-01-21T03:07:41.089 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:19.0.0-766.g0edf41a6.el9.x 9/31 2024-01-21T03:07:41.113 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:19.0.0-766.g0edf41a6.el9.x 9/31 2024-01-21T03:07:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 10/31 2024-01-21T03:07:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 11/31 2024-01-21T03:07:41.183 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 10/31 2024-01-21T03:07:41.204 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 11/31 2024-01-21T03:07:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 11/31 2024-01-21T03:07:41.253 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 11/31 2024-01-21T03:07:41.337 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-21T03:07:41.365 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-21T03:07:41.375 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-21T03:07:41.395 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-21T03:07:41.415 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-21T03:07:41.436 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-21T03:07:41.454 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-21T03:07:41.477 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-21T03:07:41.483 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-21T03:07:41.506 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-21T03:07:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-21T03:07:41.538 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-21T03:07:41.823 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-21T03:07:41.855 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 19/31 2024-01-21T03:07:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-21T03:07:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 19/31 2024-01-21T03:07:41.927 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 19/31 2024-01-21T03:07:41.959 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 20/31 2024-01-21T03:07:41.983 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 19/31 2024-01-21T03:07:42.001 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-21T03:07:42.014 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 20/31 2024-01-21T03:07:42.061 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-21T03:07:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-21T03:07:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 23/31 2024-01-21T03:07:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-21T03:07:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 23/31 2024-01-21T03:07:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 23/31 2024-01-21T03:07:42.603 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 24/31 2024-01-21T03:07:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 23/31 2024-01-21T03:07:42.634 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:19.0.0-766.g0edf41a6.el9.x86 25/31 2024-01-21T03:07:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 24/31 2024-01-21T03:07:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:19.0.0-766.g0edf41a6.el9.x86 25/31 2024-01-21T03:07:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 26/31 2024-01-21T03:07:44.814 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 26/31 2024-01-21T03:07:45.006 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 26/31 2024-01-21T03:07:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 26/31 2024-01-21T03:07:45.343 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 26/31 2024-01-21T03:07:45.384 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 27/31 2024-01-21T03:07:45.473 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 27/31 2024-01-21T03:07:45.473 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-21T03:07:45.473 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-21T03:07:45.473 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:07:45.510 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 28/31 2024-01-21T03:07:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 26/31 2024-01-21T03:07:45.647 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 27/31 2024-01-21T03:07:45.746 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 27/31 2024-01-21T03:07:45.746 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-21T03:07:45.746 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-21T03:07:45.747 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:07:45.781 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 28/31 2024-01-21T03:07:54.847 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 28/31 2024-01-21T03:07:54.847 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-01-21T03:07:54.847 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-01-21T03:07:54.847 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-01-21T03:07:54.847 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-01-21T03:07:54.847 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-01-21T03:07:54.848 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-01-21T03:07:54.848 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-01-21T03:07:54.848 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:07:55.643 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 29/31 2024-01-21T03:07:55.667 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 29/31 2024-01-21T03:07:55.667 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:07:55.668 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-21T03:07:55.668 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-21T03:07:55.668 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-21T03:07:55.668 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:07:55.669 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-21T03:07:55.753 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-21T03:07:55.753 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-21T03:07:55.813 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 28/31 2024-01-21T03:07:55.813 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2024-01-21T03:07:55.813 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2024-01-21T03:07:55.813 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2024-01-21T03:07:55.814 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2024-01-21T03:07:55.814 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2024-01-21T03:07:55.814 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /root 2024-01-21T03:07:55.814 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2024-01-21T03:07:55.814 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:07:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 29/31 2024-01-21T03:07:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 29/31 2024-01-21T03:07:56.679 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:07:56.679 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-21T03:07:56.679 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-21T03:07:56.679 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-21T03:07:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:07:56.681 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-21T03:07:56.748 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-21T03:07:56.748 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-21T03:07:56.856 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-21T03:07:56.856 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 1/31 2024-01-21T03:07:56.856 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 2/31 2024-01-21T03:07:56.856 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 3/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 4/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 5/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 6/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 7/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:19.0.0-766.g0edf41a6.el9.x 8/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:19.0.0-766.g0edf41a6.el9.x86 9/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 10/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 11/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 12/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 13/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-21T03:07:56.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-21T03:07:56.858 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-21T03:07:56.858 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-21T03:07:56.858 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-21T03:07:56.858 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-21T03:07:56.858 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-21T03:07:56.858 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-21T03:07:56.858 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-21T03:07:56.858 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-21T03:07:56.858 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 28/31 2024-01-21T03:07:56.858 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-21T03:07:56.859 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 30/31 2024-01-21T03:07:57.885 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-21T03:07:57.885 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 1/31 2024-01-21T03:07:57.885 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 2/31 2024-01-21T03:07:57.885 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 3/31 2024-01-21T03:07:57.885 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 4/31 2024-01-21T03:07:57.885 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 5/31 2024-01-21T03:07:57.885 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 6/31 2024-01-21T03:07:57.885 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 7/31 2024-01-21T03:07:57.885 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:19.0.0-766.g0edf41a6.el9.x 8/31 2024-01-21T03:07:57.885 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:19.0.0-766.g0edf41a6.el9.x86 9/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 10/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 11/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 12/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 13/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-21T03:07:57.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-21T03:07:57.887 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-21T03:07:57.887 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-21T03:07:57.887 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-21T03:07:57.887 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 28/31 2024-01-21T03:07:57.887 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-21T03:07:57.887 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 30/31 2024-01-21T03:07:58.050 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-21T03:07:58.050 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:07:58.050 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-01-21T03:07:58.050 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-21T03:07:58.051 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.052 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-21T03:07:58.052 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.052 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-21T03:07:58.052 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-21T03:07:58.052 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-21T03:07:58.052 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.052 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.052 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.052 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-21T03:07:58.052 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.052 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.052 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.052 INFO:teuthology.orchestra.run.smithi177.stdout: re2-1:20211101-3.el9.x86_64 2024-01-21T03:07:58.053 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-21T03:07:58.053 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:07:58.053 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:07:58.396 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-test 2024-01-21T03:07:58.874 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-21T03:07:58.874 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:07:58.874 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2024-01-21T03:07:58.874 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.874 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.874 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:07:58.874 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-21T03:07:58.874 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.874 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.874 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.874 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.875 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.876 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.876 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-21T03:07:58.876 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.876 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.876 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:07:58.876 INFO:teuthology.orchestra.run.smithi007.stdout: re2-1:20211101-3.el9.x86_64 2024-01-21T03:07:58.876 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-21T03:07:58.876 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:07:58.876 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:07:58.948 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:32 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:07:59.100 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:07:59.100 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:07:59.101 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-01-21T03:07:59.101 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:07:59.101 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:07:59.101 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 46 M 2024-01-21T03:07:59.101 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-01-21T03:07:59.101 INFO:teuthology.orchestra.run.smithi177.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-21T03:07:59.101 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 164 k 2024-01-21T03:07:59.101 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-21T03:07:59.101 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-21T03:07:59.101 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-21T03:07:59.101 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:07:59.101 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:07:59.101 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:07:59.102 INFO:teuthology.orchestra.run.smithi177.stdout:Install 6 Packages 2024-01-21T03:07:59.102 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:07:59.102 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 47 M 2024-01-21T03:07:59.102 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 201 M 2024-01-21T03:07:59.102 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:07:59.308 INFO:teuthology.orchestra.run.smithi177.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-01-21T03:07:59.342 INFO:teuthology.orchestra.run.smithi177.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-01-21T03:07:59.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test 2024-01-21T03:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-01-21T03:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-01-21T03:07:59.792 INFO:teuthology.orchestra.run.smithi177.stdout:(5/6): libcephsqlite-19.0.0-766.g0edf41a6.el9.x 237 kB/s | 164 kB 00:00 2024-01-21T03:07:59.920 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:33 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:00.084 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:08:00.085 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:08:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-01-21T03:08:00.085 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:08:00.085 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:08:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 46 M 2024-01-21T03:08:00.085 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-01-21T03:08:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-21T03:08:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 164 k 2024-01-21T03:08:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-21T03:08:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-21T03:08:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-21T03:08:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:00.086 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:08:00.086 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:08:00.086 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2024-01-21T03:08:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:00.087 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 47 M 2024-01-21T03:08:00.087 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 201 M 2024-01-21T03:08:00.087 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:08:00.228 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.1 MB/s | 300 kB 00:00 2024-01-21T03:08:00.295 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-01-21T03:08:00.337 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.3 MB/s | 186 kB 00:00 2024-01-21T03:08:00.379 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.2 MB/s | 217 kB 00:00 2024-01-21T03:08:00.771 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): libcephsqlite-19.0.0-766.g0edf41a6.el9.x 239 kB/s | 164 kB 00:00 2024-01-21T03:08:06.642 INFO:teuthology.orchestra.run.smithi177.stdout:(6/6): ceph-test-19.0.0-766.g0edf41a6.el9.x86_6 6.1 MB/s | 46 MB 00:07 2024-01-21T03:08:06.646 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:08:06.646 INFO:teuthology.orchestra.run.smithi177.stdout:Total 6.2 MB/s | 47 MB 00:07 2024-01-21T03:08:06.646 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:08:06.666 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:08:06.666 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:08:07.046 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:08:07.046 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:08:07.601 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:08:07.665 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-01-21T03:08:07.699 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-01-21T03:08:07.751 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-01-21T03:08:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-01-21T03:08:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 5/6 2024-01-21T03:08:12.173 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 5/6 2024-01-21T03:08:12.243 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 6/6 2024-01-21T03:08:14.576 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 6/6 2024-01-21T03:08:14.576 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 1/6 2024-01-21T03:08:14.577 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 2/6 2024-01-21T03:08:14.577 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-01-21T03:08:14.577 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-01-21T03:08:14.577 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-01-21T03:08:14.930 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-01-21T03:08:14.930 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:14.930 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:08:14.930 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:14.930 INFO:teuthology.orchestra.run.smithi177.stdout: jq-1.6-16.el9.x86_64 2024-01-21T03:08:14.930 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:14.930 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-21T03:08:14.930 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-21T03:08:14.930 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-21T03:08:14.930 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:14.930 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:08:15.246 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph 2024-01-21T03:08:15.811 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:49 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:15.965 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:08:15.968 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-01-21T03:08:15.968 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-01-21T03:08:15.968 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-01-21T03:08:15.968 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:08:15.968 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 6.5 k 2024-01-21T03:08:15.968 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-01-21T03:08:15.968 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 2.1 M 2024-01-21T03:08:15.968 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.5 M 2024-01-21T03:08:15.968 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 245 k 2024-01-21T03:08:15.968 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 4.7 M 2024-01-21T03:08:15.968 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 17 M 2024-01-21T03:08:15.968 INFO:teuthology.orchestra.run.smithi177.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-21T03:08:15.968 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-21T03:08:15.969 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:08:15.970 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-01-21T03:08:15.971 INFO:teuthology.orchestra.run.smithi177.stdout:Install 36 Packages 2024-01-21T03:08:15.971 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:15.971 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 30 M 2024-01-21T03:08:15.971 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 106 M 2024-01-21T03:08:15.971 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:08:16.621 INFO:teuthology.orchestra.run.smithi177.stdout:(1/36): ceph-19.0.0-766.g0edf41a6.el9.x86_64.rp 15 kB/s | 6.5 kB 00:00 2024-01-21T03:08:17.521 INFO:teuthology.orchestra.run.smithi177.stdout:(2/36): ceph-mgr-19.0.0-766.g0edf41a6.el9.x86_6 1.1 MB/s | 1.5 MB 00:01 2024-01-21T03:08:17.613 INFO:teuthology.orchestra.run.smithi177.stdout:(3/36): ceph-mds-19.0.0-766.g0edf41a6.el9.x86_6 1.5 MB/s | 2.1 MB 00:01 2024-01-21T03:08:17.713 INFO:teuthology.orchestra.run.smithi177.stdout:(4/36): ceph-mgr-modules-core-19.0.0-766.g0edf4 2.4 MB/s | 245 kB 00:00 2024-01-21T03:08:17.980 INFO:teuthology.orchestra.run.smithi177.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 701 kB/s | 187 kB 00:00 2024-01-21T03:08:17.994 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): ceph-test-19.0.0-766.g0edf41a6.el9.x86_6 2.6 MB/s | 46 MB 00:17 2024-01-21T03:08:17.997 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:08:17.997 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.6 MB/s | 47 MB 00:17 2024-01-21T03:08:17.998 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:08:18.020 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:08:18.020 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:08:18.055 INFO:teuthology.orchestra.run.smithi177.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-01-21T03:08:18.105 INFO:teuthology.orchestra.run.smithi177.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 633 kB/s | 32 kB 00:00 2024-01-21T03:08:18.139 INFO:teuthology.orchestra.run.smithi177.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-01-21T03:08:18.164 INFO:teuthology.orchestra.run.smithi177.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-01-21T03:08:18.223 INFO:teuthology.orchestra.run.smithi177.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-01-21T03:08:18.314 INFO:teuthology.orchestra.run.smithi177.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-01-21T03:08:18.373 INFO:teuthology.orchestra.run.smithi177.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-01-21T03:08:18.416 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:08:18.416 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:08:18.424 INFO:teuthology.orchestra.run.smithi177.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-01-21T03:08:18.474 INFO:teuthology.orchestra.run.smithi177.stdout:(14/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-01-21T03:08:18.516 INFO:teuthology.orchestra.run.smithi177.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-01-21T03:08:18.675 INFO:teuthology.orchestra.run.smithi177.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-01-21T03:08:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:08:19.079 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-01-21T03:08:19.110 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-01-21T03:08:19.164 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-01-21T03:08:19.200 INFO:teuthology.orchestra.run.smithi177.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 288 kB/s | 151 kB 00:00 2024-01-21T03:08:19.212 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-01-21T03:08:19.239 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 5/6 2024-01-21T03:08:19.333 INFO:teuthology.orchestra.run.smithi177.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 326 kB/s | 43 kB 00:00 2024-01-21T03:08:19.576 INFO:teuthology.orchestra.run.smithi177.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 710 kB/s | 172 kB 00:00 2024-01-21T03:08:19.859 INFO:teuthology.orchestra.run.smithi177.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-01-21T03:08:19.984 INFO:teuthology.orchestra.run.smithi177.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 86 kB/s | 11 kB 00:00 2024-01-21T03:08:20.368 INFO:teuthology.orchestra.run.smithi177.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 46 kB/s | 18 kB 00:00 2024-01-21T03:08:20.510 INFO:teuthology.orchestra.run.smithi177.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 163 kB/s | 23 kB 00:00 2024-01-21T03:08:20.694 INFO:teuthology.orchestra.run.smithi177.stdout:(24/36): ceph-mon-19.0.0-766.g0edf41a6.el9.x86_ 1.1 MB/s | 4.7 MB 00:04 2024-01-21T03:08:20.719 INFO:teuthology.orchestra.run.smithi177.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 93 kB/s | 19 kB 00:00 2024-01-21T03:08:20.844 INFO:teuthology.orchestra.run.smithi177.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 370 kB/s | 46 kB 00:00 2024-01-21T03:08:20.986 INFO:teuthology.orchestra.run.smithi177.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 557 kB/s | 79 kB 00:00 2024-01-21T03:08:21.011 INFO:teuthology.orchestra.run.smithi177.stdout:(28/36): python3-jaraco-text-3.2.0-6.el9.noarch 62 kB/s | 20 kB 00:00 2024-01-21T03:08:21.170 INFO:teuthology.orchestra.run.smithi177.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 104 kB/s | 16 kB 00:00 2024-01-21T03:08:21.195 INFO:teuthology.orchestra.run.smithi177.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-01-21T03:08:21.329 INFO:teuthology.orchestra.run.smithi177.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 268 kB/s | 36 kB 00:00 2024-01-21T03:08:21.437 INFO:teuthology.orchestra.run.smithi177.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 337 kB/s | 90 kB 00:00 2024-01-21T03:08:21.563 INFO:teuthology.orchestra.run.smithi177.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 983 kB/s | 230 kB 00:00 2024-01-21T03:08:21.696 INFO:teuthology.orchestra.run.smithi177.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 150 kB/s | 20 kB 00:00 2024-01-21T03:08:21.780 INFO:teuthology.orchestra.run.smithi177.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.2 MB/s | 427 kB 00:00 2024-01-21T03:08:23.180 INFO:teuthology.orchestra.run.smithi177.stdout:(36/36): ceph-osd-19.0.0-766.g0edf41a6.el9.x86_ 3.0 MB/s | 17 MB 00:05 2024-01-21T03:08:23.184 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:08:23.184 INFO:teuthology.orchestra.run.smithi177.stdout:Total 4.1 MB/s | 30 MB 00:07 2024-01-21T03:08:23.205 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:08:23.249 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:08:23.249 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:08:23.675 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:08:23.676 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:08:23.832 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 5/6 2024-01-21T03:08:23.907 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 6/6 2024-01-21T03:08:25.032 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:08:25.091 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-21T03:08:25.126 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-21T03:08:25.160 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-21T03:08:25.230 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-21T03:08:25.267 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-21T03:08:25.299 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-21T03:08:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-21T03:08:25.469 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-21T03:08:25.510 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-21T03:08:25.615 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-21T03:08:25.669 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-01-21T03:08:25.708 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2024-01-21T03:08:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-01-21T03:08:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-01-21T03:08:26.019 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-01-21T03:08:26.072 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-01-21T03:08:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-01-21T03:08:26.137 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-01-21T03:08:26.167 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-01-21T03:08:26.242 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-01-21T03:08:26.277 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-21T03:08:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 6/6 2024-01-21T03:08:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 1/6 2024-01-21T03:08:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 2/6 2024-01-21T03:08:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-01-21T03:08:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-01-21T03:08:26.302 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-01-21T03:08:26.310 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-01-21T03:08:26.350 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-01-21T03:08:26.385 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-01-21T03:08:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-01-21T03:08:26.516 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-01-21T03:08:26.559 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-01-21T03:08:26.630 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-01-21T03:08:26.716 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-01-21T03:08:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:26.717 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:08:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: jq-1.6-16.el9.x86_64 2024-01-21T03:08:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-21T03:08:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-21T03:08:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-21T03:08:26.717 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:26.717 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:08:26.766 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-01-21T03:08:26.883 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-21T03:08:26.982 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph 2024-01-21T03:08:27.042 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-766.g0edf41a6.el9.n 31/36 2024-01-21T03:08:27.064 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 32/36 2024-01-21T03:08:27.090 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 32/36 2024-01-21T03:08:27.090 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:08:27.090 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-21T03:08:27.090 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-21T03:08:27.090 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-21T03:08:27.090 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:27.552 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:01 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:27.712 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:08:27.714 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-01-21T03:08:27.714 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-01-21T03:08:27.715 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-01-21T03:08:27.715 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:08:27.715 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 6.5 k 2024-01-21T03:08:27.715 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-01-21T03:08:27.715 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 2.1 M 2024-01-21T03:08:27.715 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.5 M 2024-01-21T03:08:27.715 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 245 k 2024-01-21T03:08:27.715 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 4.7 M 2024-01-21T03:08:27.715 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 17 M 2024-01-21T03:08:27.715 INFO:teuthology.orchestra.run.smithi007.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-21T03:08:27.716 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-21T03:08:27.716 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-21T03:08:27.716 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-21T03:08:27.716 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-21T03:08:27.716 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-21T03:08:27.717 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-21T03:08:27.717 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-21T03:08:27.717 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-21T03:08:27.717 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-21T03:08:27.717 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-21T03:08:27.717 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-21T03:08:27.717 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-21T03:08:27.717 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-21T03:08:27.718 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-21T03:08:27.718 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-21T03:08:27.718 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-21T03:08:27.718 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-21T03:08:27.718 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-21T03:08:27.718 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-21T03:08:27.718 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-21T03:08:27.718 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-21T03:08:27.718 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-21T03:08:27.718 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-21T03:08:27.718 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-21T03:08:27.718 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-21T03:08:27.719 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-21T03:08:27.719 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-21T03:08:27.719 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-21T03:08:27.719 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-21T03:08:27.719 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:27.720 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:08:27.720 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-01-21T03:08:27.720 INFO:teuthology.orchestra.run.smithi007.stdout:Install 36 Packages 2024-01-21T03:08:27.720 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:27.721 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 30 M 2024-01-21T03:08:27.721 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 106 M 2024-01-21T03:08:27.721 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:08:28.387 INFO:teuthology.orchestra.run.smithi007.stdout:(1/36): ceph-19.0.0-766.g0edf41a6.el9.x86_64.rp 29 kB/s | 6.5 kB 00:00 2024-01-21T03:08:28.531 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 33/36 2024-01-21T03:08:28.559 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 33/36 2024-01-21T03:08:28.559 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:08:28.559 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-21T03:08:28.559 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-21T03:08:28.559 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-21T03:08:28.559 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:28.998 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 34/36 2024-01-21T03:08:29.004 INFO:teuthology.orchestra.run.smithi007.stdout:(2/36): ceph-mgr-19.0.0-766.g0edf41a6.el9.x86_6 1.8 MB/s | 1.5 MB 00:00 2024-01-21T03:08:29.022 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 34/36 2024-01-21T03:08:29.022 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:08:29.022 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-21T03:08:29.022 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-21T03:08:29.022 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-21T03:08:29.022 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:29.232 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 35/36 2024-01-21T03:08:29.256 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 35/36 2024-01-21T03:08:29.256 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:08:29.256 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-21T03:08:29.256 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-21T03:08:29.256 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-21T03:08:29.256 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:29.296 INFO:teuthology.orchestra.run.smithi007.stdout:(3/36): ceph-mds-19.0.0-766.g0edf41a6.el9.x86_6 1.9 MB/s | 2.1 MB 00:01 2024-01-21T03:08:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 36/36 2024-01-21T03:08:29.430 INFO:teuthology.orchestra.run.smithi007.stdout:(4/36): ceph-mgr-modules-core-19.0.0-766.g0edf4 1.8 MB/s | 245 kB 00:00 2024-01-21T03:08:29.830 INFO:teuthology.orchestra.run.smithi007.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 467 kB/s | 187 kB 00:00 2024-01-21T03:08:29.897 INFO:teuthology.orchestra.run.smithi007.stdout:(6/36): ceph-mon-19.0.0-766.g0edf41a6.el9.x86_6 3.1 MB/s | 4.7 MB 00:01 2024-01-21T03:08:29.972 INFO:teuthology.orchestra.run.smithi007.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 144 kB 00:00 2024-01-21T03:08:29.989 INFO:teuthology.orchestra.run.smithi007.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 344 kB/s | 32 kB 00:00 2024-01-21T03:08:30.023 INFO:teuthology.orchestra.run.smithi007.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 944 kB/s | 47 kB 00:00 2024-01-21T03:08:30.040 INFO:teuthology.orchestra.run.smithi007.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 767 kB/s | 39 kB 00:00 2024-01-21T03:08:30.123 INFO:teuthology.orchestra.run.smithi007.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.4 MB/s | 241 kB 00:00 2024-01-21T03:08:30.207 INFO:teuthology.orchestra.run.smithi007.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 6.9 MB/s | 1.1 MB 00:00 2024-01-21T03:08:30.232 INFO:teuthology.orchestra.run.smithi007.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 948 kB/s | 103 kB 00:00 2024-01-21T03:08:30.257 INFO:teuthology.orchestra.run.smithi007.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-01-21T03:08:30.283 INFO:teuthology.orchestra.run.smithi007.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-01-21T03:08:30.308 INFO:teuthology.orchestra.run.smithi007.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-01-21T03:08:30.475 INFO:teuthology.orchestra.run.smithi007.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 906 kB/s | 151 kB 00:00 2024-01-21T03:08:30.508 INFO:teuthology.orchestra.run.smithi007.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-01-21T03:08:30.584 INFO:teuthology.orchestra.run.smithi007.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-01-21T03:08:30.601 INFO:teuthology.orchestra.run.smithi007.stdout:(20/36): lua-devel-5.4.4-4.el9.x86_64.rpm 70 kB/s | 22 kB 00:00 2024-01-21T03:08:30.651 INFO:teuthology.orchestra.run.smithi007.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-01-21T03:08:30.684 INFO:teuthology.orchestra.run.smithi007.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 532 kB/s | 18 kB 00:00 2024-01-21T03:08:30.710 INFO:teuthology.orchestra.run.smithi007.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-01-21T03:08:30.727 INFO:teuthology.orchestra.run.smithi007.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 553 kB/s | 23 kB 00:00 2024-01-21T03:08:30.744 INFO:teuthology.orchestra.run.smithi007.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 579 kB/s | 19 kB 00:00 2024-01-21T03:08:30.761 INFO:teuthology.orchestra.run.smithi007.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 584 kB/s | 20 kB 00:00 2024-01-21T03:08:30.778 INFO:teuthology.orchestra.run.smithi007.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-01-21T03:08:30.954 INFO:teuthology.orchestra.run.smithi007.stdout:(28/36): ceph-osd-19.0.0-766.g0edf41a6.el9.x86_ 8.6 MB/s | 17 MB 00:01 2024-01-21T03:08:30.979 INFO:teuthology.orchestra.run.smithi007.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 362 kB/s | 79 kB 00:00 2024-01-21T03:08:31.004 INFO:teuthology.orchestra.run.smithi007.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-01-21T03:08:31.071 INFO:teuthology.orchestra.run.smithi007.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 536 kB/s | 36 kB 00:00 2024-01-21T03:08:31.085 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 36/36 2024-01-21T03:08:31.085 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 1/36 2024-01-21T03:08:31.085 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 2/36 2024-01-21T03:08:31.085 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 3/36 2024-01-21T03:08:31.085 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 4/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 5/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-766.g0edf41a6.el9.n 6/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-21T03:08:31.086 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-21T03:08:31.087 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-21T03:08:31.087 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-21T03:08:31.087 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-21T03:08:31.087 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-21T03:08:31.087 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-21T03:08:31.087 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-21T03:08:31.088 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-21T03:08:31.088 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-21T03:08:31.088 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-21T03:08:31.088 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-21T03:08:31.089 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-21T03:08:31.089 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-21T03:08:31.089 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-21T03:08:31.089 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-21T03:08:31.089 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-21T03:08:31.089 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-21T03:08:31.089 INFO:teuthology.orchestra.run.smithi007.stdout:(32/36): python3-portend-3.1.0-2.el9.noarch.rpm 123 kB/s | 16 kB 00:00 2024-01-21T03:08:31.113 INFO:teuthology.orchestra.run.smithi007.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 671 kB/s | 90 kB 00:00 2024-01-21T03:08:31.188 INFO:teuthology.orchestra.run.smithi007.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 267 kB/s | 20 kB 00:00 2024-01-21T03:08:31.247 INFO:teuthology.orchestra.run.smithi007.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-01-21T03:08:31.405 INFO:teuthology.orchestra.run.smithi007.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.3 MB/s | 427 kB 00:00 2024-01-21T03:08:31.409 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:08:31.409 INFO:teuthology.orchestra.run.smithi007.stdout:Total 8.0 MB/s | 30 MB 00:03 2024-01-21T03:08:31.432 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:08:31.479 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:08:31.479 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:08:31.901 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:08:31.901 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:08:32.156 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-21T03:08:32.156 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:32.156 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:08:32.156 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:32.156 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:32.156 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: lua-5.4.4-4.el9.x86_64 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-21T03:08:32.157 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-21T03:08:32.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-21T03:08:32.159 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-21T03:08:32.159 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-21T03:08:32.159 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-21T03:08:32.159 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:32.159 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:08:32.488 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base 2024-01-21T03:08:33.046 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:07 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:33.150 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:08:33.200 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:08:33.201 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-01-21T03:08:33.201 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:08:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:08:33.270 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install cephadm 2024-01-21T03:08:33.300 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-21T03:08:33.340 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-21T03:08:33.369 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-21T03:08:33.443 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-21T03:08:33.485 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-21T03:08:33.517 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-21T03:08:33.612 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-21T03:08:33.699 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-21T03:08:33.745 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-21T03:08:33.819 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-21T03:08:33.823 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:07 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:33.874 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-01-21T03:08:33.912 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2024-01-21T03:08:33.962 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-01-21T03:08:33.973 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:08:33.974 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:08:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-01-21T03:08:33.974 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:08:33.974 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:08:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 742 k 2024-01-21T03:08:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:33.974 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:08:33.974 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:08:33.974 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:08:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:33.974 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 742 k 2024-01-21T03:08:33.974 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 748 k 2024-01-21T03:08:33.974 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:08:34.017 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-01-21T03:08:34.220 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-01-21T03:08:34.266 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-01-21T03:08:34.304 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-01-21T03:08:34.339 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-01-21T03:08:34.369 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-01-21T03:08:34.448 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-01-21T03:08:34.488 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-21T03:08:34.521 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-01-21T03:08:34.562 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-01-21T03:08:34.597 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-01-21T03:08:34.606 INFO:teuthology.orchestra.run.smithi177.stdout:cephadm-19.0.0-766.g0edf41a6.el9.noarch.rpm 1.1 MB/s | 742 kB 00:00 2024-01-21T03:08:34.606 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:08:34.607 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.1 MB/s | 742 kB 00:00 2024-01-21T03:08:34.607 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:08:34.612 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:08:34.612 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:08:34.618 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:08:34.618 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:08:34.631 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-01-21T03:08:34.695 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:08:34.725 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-01-21T03:08:34.762 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-01-21T03:08:34.836 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-01-21T03:08:34.977 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-01-21T03:08:35.103 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-21T03:08:35.200 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:08:35.272 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-766.g0edf41a6.el9.n 31/36 2024-01-21T03:08:35.292 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:08:35.294 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 32/36 2024-01-21T03:08:35.320 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 32/36 2024-01-21T03:08:35.320 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:08:35.320 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-21T03:08:35.320 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-21T03:08:35.320 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-21T03:08:35.320 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:35.689 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:08:35.978 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:08:35.978 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:35.978 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:08:35.978 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:08:35.978 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:35.978 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:08:36.185 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T03:08:36.747 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:10 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:36.845 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 33/36 2024-01-21T03:08:36.871 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 33/36 2024-01-21T03:08:36.871 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:08:36.871 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-21T03:08:36.871 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-21T03:08:36.871 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-21T03:08:36.872 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:36.900 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:08:36.901 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:08:36.901 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:08:36.901 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:08:36.901 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:08:36.901 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 145 k 2024-01-21T03:08:36.901 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:36.901 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:08:36.902 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:08:36.902 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:08:36.902 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:36.902 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 145 k 2024-01-21T03:08:36.902 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 436 k 2024-01-21T03:08:36.902 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:08:37.338 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 34/36 2024-01-21T03:08:37.344 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-immutable-object-cache-19.0.0-766.g0edf41a 329 kB/s | 145 kB 00:00 2024-01-21T03:08:37.345 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:08:37.345 INFO:teuthology.orchestra.run.smithi177.stdout:Total 328 kB/s | 145 kB 00:00 2024-01-21T03:08:37.345 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:08:37.354 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:08:37.354 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:08:37.361 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 34/36 2024-01-21T03:08:37.361 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:08:37.361 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-21T03:08:37.361 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-21T03:08:37.361 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-21T03:08:37.361 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:37.405 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:08:37.406 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:08:37.549 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:08:37.580 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 35/36 2024-01-21T03:08:37.602 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 35/36 2024-01-21T03:08:37.603 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:08:37.603 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-21T03:08:37.603 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-21T03:08:37.603 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-21T03:08:37.603 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:37.628 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.e 1/1 2024-01-21T03:08:37.653 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.e 1/1 2024-01-21T03:08:37.653 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:08:37.653 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-21T03:08:37.653 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:37.671 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 36/36 2024-01-21T03:08:38.683 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.e 1/1 2024-01-21T03:08:38.683 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:38.683 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:08:38.683 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:38.683 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:38.683 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:08:38.907 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr 2024-01-21T03:08:39.477 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:13 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:39.505 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 36/36 2024-01-21T03:08:39.505 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 1/36 2024-01-21T03:08:39.505 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 2/36 2024-01-21T03:08:39.505 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 3/36 2024-01-21T03:08:39.505 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 4/36 2024-01-21T03:08:39.505 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 5/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-766.g0edf41a6.el9.n 6/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-21T03:08:39.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-21T03:08:39.507 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-21T03:08:39.507 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-21T03:08:39.507 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-21T03:08:39.507 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-21T03:08:39.507 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-21T03:08:39.508 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-21T03:08:39.508 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-21T03:08:39.508 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-21T03:08:39.508 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-21T03:08:39.508 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-21T03:08:39.508 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-21T03:08:39.508 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-21T03:08:39.509 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-21T03:08:39.578 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:08:39.628 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:08:39.629 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-01-21T03:08:39.629 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:08:39.705 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T03:08:40.259 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:14 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:40.410 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:08:40.411 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:08:40.411 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-01-21T03:08:40.411 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:08:40.411 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:08:40.411 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 1.7 M 2024-01-21T03:08:40.411 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-01-21T03:08:40.412 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 28 k 2024-01-21T03:08:40.412 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 15 k 2024-01-21T03:08:40.412 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-21T03:08:40.412 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-21T03:08:40.412 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:40.412 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:08:40.412 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:08:40.412 INFO:teuthology.orchestra.run.smithi177.stdout:Install 5 Packages 2024-01-21T03:08:40.412 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:40.413 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 2.0 M 2024-01-21T03:08:40.413 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 8.5 M 2024-01-21T03:08:40.413 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:08:40.523 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-21T03:08:40.523 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:40.523 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:08:40.523 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:40.523 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:40.523 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:40.523 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:08:40.523 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:40.523 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:40.523 INFO:teuthology.orchestra.run.smithi007.stdout: lua-5.4.4-4.el9.x86_64 2024-01-21T03:08:40.524 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-21T03:08:40.524 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-21T03:08:40.524 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-21T03:08:40.524 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-21T03:08:40.524 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-21T03:08:40.524 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-21T03:08:40.524 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-21T03:08:40.524 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-21T03:08:40.524 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-21T03:08:40.524 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-21T03:08:40.524 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-21T03:08:40.524 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-21T03:08:40.524 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply-3.11-14.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-21T03:08:40.525 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-21T03:08:40.526 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-21T03:08:40.526 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:40.526 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:08:40.814 INFO:teuthology.orchestra.run.smithi177.stdout:(1/5): ceph-grafana-dashboards-19.0.0-766.g0edf 146 kB/s | 28 kB 00:00 2024-01-21T03:08:40.839 INFO:teuthology.orchestra.run.smithi177.stdout:(2/5): ceph-prometheus-alerts-19.0.0-766.g0edf4 69 kB/s | 15 kB 00:00 2024-01-21T03:08:40.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base 2024-01-21T03:08:41.098 INFO:teuthology.orchestra.run.smithi177.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 108 kB/s | 31 kB 00:00 2024-01-21T03:08:41.248 INFO:teuthology.orchestra.run.smithi177.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 462 kB/s | 188 kB 00:00 2024-01-21T03:08:41.489 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:15 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:41.593 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:08:41.623 INFO:teuthology.orchestra.run.smithi177.stdout:(5/5): ceph-mgr-dashboard-19.0.0-766.g0edf41a6. 1.7 MB/s | 1.7 MB 00:01 2024-01-21T03:08:41.627 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:08:41.627 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-01-21T03:08:41.627 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:08:41.634 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:08:41.634 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:08:41.645 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:08:41.647 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-01-21T03:08:41.647 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:08:41.694 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:08:41.694 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:08:41.717 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install cephadm 2024-01-21T03:08:41.946 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:08:42.029 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-21T03:08:42.063 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-21T03:08:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-766.g0edf41a6.el9.no 3/5 2024-01-21T03:08:42.284 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:16 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-766.g0edf41a6.el9.n 4/5 2024-01-21T03:08:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 5/5 2024-01-21T03:08:42.439 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:08:42.440 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:08:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-01-21T03:08:42.440 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:08:42.440 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:08:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 742 k 2024-01-21T03:08:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:42.441 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:08:42.441 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:08:42.441 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:08:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:42.441 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 742 k 2024-01-21T03:08:42.441 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 748 k 2024-01-21T03:08:42.441 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:08:42.787 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm-19.0.0-766.g0edf41a6.el9.noarch.rpm 2.1 MB/s | 742 kB 00:00 2024-01-21T03:08:42.788 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:08:42.788 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.1 MB/s | 742 kB 00:00 2024-01-21T03:08:42.788 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:08:42.794 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:08:42.794 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:08:42.800 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:08:42.800 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:08:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:08:42.923 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 5/5 2024-01-21T03:08:42.923 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-766.g0edf41a6.el9.n 1/5 2024-01-21T03:08:42.923 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 2/5 2024-01-21T03:08:42.923 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-766.g0edf41a6.el9.no 3/5 2024-01-21T03:08:42.923 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-21T03:08:43.251 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-21T03:08:43.251 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:43.251 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:08:43.251 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:08:43.251 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:08:43.251 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:08:43.252 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-21T03:08:43.252 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-21T03:08:43.252 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:43.252 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:08:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:08:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:08:43.455 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T03:08:43.795 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:08:44.051 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:18 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:44.103 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:08:44.103 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:44.103 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:08:44.103 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:08:44.103 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:44.103 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:08:44.202 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:08:44.203 INFO:teuthology.orchestra.run.smithi177.stdout:=========================================================================================== 2024-01-21T03:08:44.203 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-01-21T03:08:44.203 INFO:teuthology.orchestra.run.smithi177.stdout:=========================================================================================== 2024-01-21T03:08:44.204 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:08:44.204 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 7.4 M 2024-01-21T03:08:44.204 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-01-21T03:08:44.204 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-21T03:08:44.204 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-21T03:08:44.204 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-21T03:08:44.204 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-21T03:08:44.204 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-21T03:08:44.204 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-21T03:08:44.204 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-21T03:08:44.204 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-21T03:08:44.204 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-21T03:08:44.204 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-21T03:08:44.205 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-21T03:08:44.205 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:44.205 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:08:44.205 INFO:teuthology.orchestra.run.smithi177.stdout:=========================================================================================== 2024-01-21T03:08:44.205 INFO:teuthology.orchestra.run.smithi177.stdout:Install 12 Packages 2024-01-21T03:08:44.205 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:44.205 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 37 M 2024-01-21T03:08:44.205 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 209 M 2024-01-21T03:08:44.205 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:08:44.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T03:08:44.424 INFO:teuthology.orchestra.run.smithi177.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 206 kB/s | 32 kB 00:00 2024-01-21T03:08:44.465 INFO:teuthology.orchestra.run.smithi177.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 414 kB/s | 17 kB 00:00 2024-01-21T03:08:44.507 INFO:teuthology.orchestra.run.smithi177.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 823 kB/s | 34 kB 00:00 2024-01-21T03:08:44.741 INFO:teuthology.orchestra.run.smithi177.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.3 MB/s | 3.0 MB 00:00 2024-01-21T03:08:44.855 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:18 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:44.967 INFO:teuthology.orchestra.run.smithi177.stdout:(5/12): ceph-mgr-diskprediction-local-19.0.0-76 11 MB/s | 7.4 MB 00:00 2024-01-21T03:08:45.013 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:08:45.013 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:08:45.014 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:08:45.014 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:08:45.014 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:08:45.014 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 145 k 2024-01-21T03:08:45.014 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:45.014 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:08:45.014 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:08:45.014 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:08:45.014 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:45.015 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 145 k 2024-01-21T03:08:45.015 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 436 k 2024-01-21T03:08:45.015 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:08:45.517 INFO:teuthology.orchestra.run.smithi177.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 396 kB/s | 217 kB 00:00 2024-01-21T03:08:46.035 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-19.0.0-766.g0edf41a 142 kB/s | 145 kB 00:01 2024-01-21T03:08:46.036 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:08:46.036 INFO:teuthology.orchestra.run.smithi007.stdout:Total 142 kB/s | 145 kB 00:01 2024-01-21T03:08:46.036 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:08:46.046 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:08:46.046 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:08:46.104 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:08:46.104 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:08:46.247 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:08:46.311 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.e 1/1 2024-01-21T03:08:46.335 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.e 1/1 2024-01-21T03:08:46.335 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:08:46.335 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-21T03:08:46.335 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:46.992 INFO:teuthology.orchestra.run.smithi177.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 1.9 MB/s | 4.7 MB 00:02 2024-01-21T03:08:47.375 INFO:teuthology.orchestra.run.smithi177.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.8 MB/s | 4.8 MB 00:02 2024-01-21T03:08:47.409 INFO:teuthology.orchestra.run.smithi177.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 802 kB 00:00 2024-01-21T03:08:47.473 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.e 1/1 2024-01-21T03:08:47.473 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:47.473 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:08:47.474 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:08:47.474 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:47.474 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:08:47.584 INFO:teuthology.orchestra.run.smithi177.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 934 kB/s | 194 kB 00:00 2024-01-21T03:08:47.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr 2024-01-21T03:08:47.726 INFO:teuthology.orchestra.run.smithi177.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 780 kB/s | 245 kB 00:00 2024-01-21T03:08:47.885 INFO:teuthology.orchestra.run.smithi177.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.6 MB/s | 16 MB 00:02 2024-01-21T03:08:47.888 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:08:47.888 INFO:teuthology.orchestra.run.smithi177.stdout:Total 10 MB/s | 37 MB 00:03 2024-01-21T03:08:47.903 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:08:47.929 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:08:47.929 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:08:48.285 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:22 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:48.319 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:08:48.319 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:08:48.388 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:08:48.440 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:08:48.441 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-01-21T03:08:48.441 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:08:48.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T03:08:49.016 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:08:49.074 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:23 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-21T03:08:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-21T03:08:49.231 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:08:49.231 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-21T03:08:49.232 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:08:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-01-21T03:08:49.232 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:08:49.232 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:08:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 1.7 M 2024-01-21T03:08:49.232 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-01-21T03:08:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 28 k 2024-01-21T03:08:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 15 k 2024-01-21T03:08:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-21T03:08:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-21T03:08:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:49.233 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:08:49.233 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:08:49.233 INFO:teuthology.orchestra.run.smithi007.stdout:Install 5 Packages 2024-01-21T03:08:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:49.233 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.0 M 2024-01-21T03:08:49.233 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 8.5 M 2024-01-21T03:08:49.233 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:08:49.267 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-21T03:08:49.454 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-21T03:08:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-21T03:08:49.559 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-21T03:08:49.756 INFO:teuthology.orchestra.run.smithi007.stdout:(1/5): ceph-prometheus-alerts-19.0.0-766.g0edf4 83 kB/s | 15 kB 00:00 2024-01-21T03:08:49.848 INFO:teuthology.orchestra.run.smithi007.stdout:(2/5): ceph-grafana-dashboards-19.0.0-766.g0edf 103 kB/s | 28 kB 00:00 2024-01-21T03:08:49.940 INFO:teuthology.orchestra.run.smithi007.stdout:(3/5): ceph-mgr-dashboard-19.0.0-766.g0edf41a6. 4.7 MB/s | 1.7 MB 00:00 2024-01-21T03:08:50.066 INFO:teuthology.orchestra.run.smithi007.stdout:(4/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 100 kB/s | 31 kB 00:00 2024-01-21T03:08:50.212 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-21T03:08:50.327 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-21T03:08:50.666 INFO:teuthology.orchestra.run.smithi007.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 231 kB/s | 188 kB 00:00 2024-01-21T03:08:50.667 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:08:50.668 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-01-21T03:08:50.668 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:08:50.676 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:08:50.677 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:08:50.745 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:08:50.745 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:08:50.990 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:08:51.089 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-21T03:08:51.129 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-21T03:08:51.159 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-766.g0edf41a6.el9.no 3/5 2024-01-21T03:08:51.405 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-766.g0edf41a6.el9.n 4/5 2024-01-21T03:08:51.444 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 5/5 2024-01-21T03:08:51.588 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-21T03:08:52.001 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 5/5 2024-01-21T03:08:52.002 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-766.g0edf41a6.el9.n 1/5 2024-01-21T03:08:52.002 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 2/5 2024-01-21T03:08:52.002 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-766.g0edf41a6.el9.no 3/5 2024-01-21T03:08:52.002 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-21T03:08:52.302 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-21T03:08:52.302 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:52.302 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:08:52.302 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:08:52.302 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:08:52.302 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:08:52.302 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-21T03:08:52.302 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-21T03:08:52.302 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:52.302 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:08:52.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T03:08:52.625 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-21T03:08:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41 12/12 2024-01-21T03:08:53.082 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:27 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:53.240 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:08:53.242 INFO:teuthology.orchestra.run.smithi007.stdout:=========================================================================================== 2024-01-21T03:08:53.242 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-01-21T03:08:53.242 INFO:teuthology.orchestra.run.smithi007.stdout:=========================================================================================== 2024-01-21T03:08:53.242 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:08:53.242 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 7.4 M 2024-01-21T03:08:53.242 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-01-21T03:08:53.242 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-21T03:08:53.242 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-21T03:08:53.242 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-21T03:08:53.242 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-21T03:08:53.242 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-21T03:08:53.243 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-21T03:08:53.243 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-21T03:08:53.243 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-21T03:08:53.243 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-21T03:08:53.243 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-21T03:08:53.243 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-21T03:08:53.243 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:53.243 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:08:53.243 INFO:teuthology.orchestra.run.smithi007.stdout:=========================================================================================== 2024-01-21T03:08:53.243 INFO:teuthology.orchestra.run.smithi007.stdout:Install 12 Packages 2024-01-21T03:08:53.243 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:08:53.244 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 37 M 2024-01-21T03:08:53.244 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 209 M 2024-01-21T03:08:53.244 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:08:53.514 INFO:teuthology.orchestra.run.smithi007.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 225 kB/s | 32 kB 00:00 2024-01-21T03:08:53.539 INFO:teuthology.orchestra.run.smithi007.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 730 kB/s | 17 kB 00:00 2024-01-21T03:08:53.564 INFO:teuthology.orchestra.run.smithi007.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2024-01-21T03:08:53.623 INFO:teuthology.orchestra.run.smithi007.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-01-21T03:08:53.782 INFO:teuthology.orchestra.run.smithi007.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-01-21T03:08:53.849 INFO:teuthology.orchestra.run.smithi007.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-01-21T03:08:53.873 INFO:teuthology.orchestra.run.smithi007.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-01-21T03:08:54.041 INFO:teuthology.orchestra.run.smithi007.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.8 MB/s | 802 kB 00:00 2024-01-21T03:08:54.216 INFO:teuthology.orchestra.run.smithi007.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-01-21T03:08:54.241 INFO:teuthology.orchestra.run.smithi007.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 980 kB/s | 194 kB 00:00 2024-01-21T03:08:54.342 INFO:teuthology.orchestra.run.smithi007.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-7 7.6 MB/s | 7.4 MB 00:00 2024-01-21T03:08:54.542 INFO:teuthology.orchestra.run.smithi007.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 755 kB/s | 245 kB 00:00 2024-01-21T03:08:54.545 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:08:54.545 INFO:teuthology.orchestra.run.smithi007.stdout:Total 28 MB/s | 37 MB 00:01 2024-01-21T03:08:54.558 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:08:54.585 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:08:54.585 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:08:54.988 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:08:54.989 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:08:55.173 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41 12/12 2024-01-21T03:08:55.173 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41 1/12 2024-01-21T03:08:55.173 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-21T03:08:55.174 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-21T03:08:55.174 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-21T03:08:55.174 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-21T03:08:55.174 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-21T03:08:55.174 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-21T03:08:55.174 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-21T03:08:55.174 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-21T03:08:55.174 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-21T03:08:55.174 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-21T03:08:55.652 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-21T03:08:55.652 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:55.652 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:55.653 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:08:55.707 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:08:55.766 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-21T03:08:55.797 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-21T03:08:55.899 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-21T03:08:55.950 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-21T03:08:56.036 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-rook 2024-01-21T03:08:56.146 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-21T03:08:56.182 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-21T03:08:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-21T03:08:56.602 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:30 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:08:56.752 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:08:56.754 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-01-21T03:08:56.754 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-01-21T03:08:56.754 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-01-21T03:08:56.754 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:08:56.754 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 51 k 2024-01-21T03:08:56.754 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-01-21T03:08:56.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-21T03:08:56.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-21T03:08:56.754 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout:Install 13 Packages 2024-01-21T03:08:56.755 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:08:56.756 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 2.0 M 2024-01-21T03:08:56.756 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 26 M 2024-01-21T03:08:56.756 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:08:56.981 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-21T03:08:57.064 INFO:teuthology.orchestra.run.smithi177.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 121 kB/s | 24 kB 00:00 2024-01-21T03:08:57.081 INFO:teuthology.orchestra.run.smithi177.stdout:(2/13): ceph-mgr-rook-19.0.0-766.g0edf41a6.el9. 232 kB/s | 51 kB 00:00 2024-01-21T03:08:57.091 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-21T03:08:57.131 INFO:teuthology.orchestra.run.smithi177.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 67 kB/s | 18 kB 00:00 2024-01-21T03:08:57.240 INFO:teuthology.orchestra.run.smithi177.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 995 kB/s | 174 kB 00:00 2024-01-21T03:08:57.290 INFO:teuthology.orchestra.run.smithi177.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 633 kB/s | 132 kB 00:00 2024-01-21T03:08:57.432 INFO:teuthology.orchestra.run.smithi177.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 700 kB/s | 211 kB 00:00 2024-01-21T03:08:57.457 INFO:teuthology.orchestra.run.smithi177.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 197 kB/s | 43 kB 00:00 2024-01-21T03:08:57.624 INFO:teuthology.orchestra.run.smithi177.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 96 kB/s | 32 kB 00:00 2024-01-21T03:08:57.699 INFO:teuthology.orchestra.run.smithi177.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-01-21T03:08:57.924 INFO:teuthology.orchestra.run.smithi177.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 263 kB/s | 59 kB 00:00 2024-01-21T03:08:57.950 INFO:teuthology.orchestra.run.smithi177.stdout:(11/13): python3-google-auth-2.26.0-1.el9.noarc 410 kB/s | 202 kB 00:00 2024-01-21T03:08:58.108 INFO:teuthology.orchestra.run.smithi177.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 489 kB/s | 90 kB 00:00 2024-01-21T03:08:58.142 INFO:teuthology.orchestra.run.smithi177.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-01-21T03:08:58.144 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:08:58.144 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-01-21T03:08:58.144 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:08:58.158 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:08:58.158 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:08:58.259 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:08:58.259 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:08:58.416 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-21T03:08:58.775 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:08:58.959 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-21T03:08:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-21T03:08:59.073 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-21T03:08:59.111 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-21T03:08:59.152 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-21T03:08:59.247 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-21T03:08:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-21T03:08:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-21T03:08:59.487 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-21T03:08:59.515 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41 12/12 2024-01-21T03:09:00.150 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-21T03:09:00.200 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-21T03:09:00.234 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-21T03:09:00.291 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-21T03:09:00.313 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 13/13 2024-01-21T03:09:01.004 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 13/13 2024-01-21T03:09:01.005 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 1/13 2024-01-21T03:09:01.005 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-21T03:09:01.005 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-21T03:09:01.005 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-21T03:09:01.005 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-21T03:09:01.005 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-21T03:09:01.005 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-21T03:09:01.005 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-21T03:09:01.005 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-21T03:09:01.005 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-21T03:09:01.005 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-21T03:09:01.005 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-21T03:09:01.390 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-21T03:09:01.390 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:01.390 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:09:01.390 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:09:01.390 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-21T03:09:01.390 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-21T03:09:01.390 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-21T03:09:01.390 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-21T03:09:01.390 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-21T03:09:01.391 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-21T03:09:01.391 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-21T03:09:01.391 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-21T03:09:01.391 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-21T03:09:01.391 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-21T03:09:01.391 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-21T03:09:01.391 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-21T03:09:01.391 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:01.391 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:01.644 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T03:09:02.083 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41 12/12 2024-01-21T03:09:02.083 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41 1/12 2024-01-21T03:09:02.083 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-21T03:09:02.083 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-21T03:09:02.083 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-21T03:09:02.083 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-21T03:09:02.083 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-21T03:09:02.083 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-21T03:09:02.083 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-21T03:09:02.083 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-21T03:09:02.083 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-21T03:09:02.083 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-21T03:09:02.217 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:36 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:02.369 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:02.370 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-01-21T03:09:02.370 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-01-21T03:09:02.370 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-01-21T03:09:02.370 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:09:02.370 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 133 k 2024-01-21T03:09:02.370 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-01-21T03:09:02.370 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-21T03:09:02.370 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-21T03:09:02.371 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-21T03:09:02.371 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-21T03:09:02.371 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-21T03:09:02.371 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:02.371 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:09:02.371 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-01-21T03:09:02.371 INFO:teuthology.orchestra.run.smithi177.stdout:Install 6 Packages 2024-01-21T03:09:02.371 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:02.371 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 6.8 M 2024-01-21T03:09:02.372 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 33 M 2024-01-21T03:09:02.372 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:09:02.655 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-21T03:09:02.655 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:02.655 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:09:02.656 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:09:02.656 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-21T03:09:02.656 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-21T03:09:02.656 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-21T03:09:02.656 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-21T03:09:02.656 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-21T03:09:02.656 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-21T03:09:02.656 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-21T03:09:02.656 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-21T03:09:02.656 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-21T03:09:02.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-21T03:09:02.657 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-21T03:09:02.657 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:02.657 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:02.819 INFO:teuthology.orchestra.run.smithi177.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2024-01-21T03:09:02.919 INFO:teuthology.orchestra.run.smithi177.stdout:(2/6): ceph-mgr-cephadm-19.0.0-766.g0edf41a6.el 481 kB/s | 133 kB 00:00 2024-01-21T03:09:03.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-rook 2024-01-21T03:09:03.227 INFO:teuthology.orchestra.run.smithi177.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 187 kB/s | 58 kB 00:00 2024-01-21T03:09:03.319 INFO:teuthology.orchestra.run.smithi177.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 560 kB/s | 51 kB 00:00 2024-01-21T03:09:03.461 INFO:teuthology.orchestra.run.smithi177.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.1 MB/s | 5.8 MB 00:00 2024-01-21T03:09:03.486 INFO:teuthology.orchestra.run.smithi177.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 821 kB/s | 548 kB 00:00 2024-01-21T03:09:03.490 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:03.490 INFO:teuthology.orchestra.run.smithi177.stdout:Total 6.1 MB/s | 6.8 MB 00:01 2024-01-21T03:09:03.490 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:09:03.499 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:09:03.499 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:09:03.567 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:37 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:03.588 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:09:03.589 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:09:03.727 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:03.728 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-01-21T03:09:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-01-21T03:09:03.729 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-01-21T03:09:03.729 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:09:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 51 k 2024-01-21T03:09:03.729 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-01-21T03:09:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-21T03:09:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-21T03:09:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-21T03:09:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-21T03:09:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-21T03:09:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-21T03:09:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-21T03:09:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-21T03:09:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-21T03:09:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-21T03:09:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-21T03:09:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-21T03:09:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:03.730 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:09:03.730 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-01-21T03:09:03.730 INFO:teuthology.orchestra.run.smithi007.stdout:Install 13 Packages 2024-01-21T03:09:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:03.731 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.0 M 2024-01-21T03:09:03.731 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 26 M 2024-01-21T03:09:03.731 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:09:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:09:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-21T03:09:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-21T03:09:04.287 INFO:teuthology.orchestra.run.smithi007.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 141 kB/s | 24 kB 00:00 2024-01-21T03:09:04.312 INFO:teuthology.orchestra.run.smithi007.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 91 kB/s | 18 kB 00:00 2024-01-21T03:09:04.371 INFO:teuthology.orchestra.run.smithi007.stdout:(3/13): ceph-mgr-rook-19.0.0-766.g0edf41a6.el9. 198 kB/s | 51 kB 00:00 2024-01-21T03:09:04.405 INFO:teuthology.orchestra.run.smithi007.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 174 kB 00:00 2024-01-21T03:09:04.430 INFO:teuthology.orchestra.run.smithi007.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 132 kB 00:00 2024-01-21T03:09:04.463 INFO:teuthology.orchestra.run.smithi007.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.2 MB/s | 211 kB 00:00 2024-01-21T03:09:04.489 INFO:teuthology.orchestra.run.smithi007.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 511 kB/s | 43 kB 00:00 2024-01-21T03:09:04.606 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-21T03:09:04.631 INFO:teuthology.orchestra.run.smithi007.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 160 kB/s | 32 kB 00:00 2024-01-21T03:09:04.656 INFO:teuthology.orchestra.run.smithi007.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2024-01-21T03:09:04.689 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-21T03:09:04.750 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-21T03:09:04.775 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 6/6 2024-01-21T03:09:04.914 INFO:teuthology.orchestra.run.smithi007.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 229 kB/s | 59 kB 00:00 2024-01-21T03:09:05.073 INFO:teuthology.orchestra.run.smithi007.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 567 kB/s | 90 kB 00:00 2024-01-21T03:09:05.098 INFO:teuthology.orchestra.run.smithi007.stdout:(12/13): python3-google-auth-2.26.0-1.el9.noarc 331 kB/s | 202 kB 00:00 2024-01-21T03:09:05.582 INFO:teuthology.orchestra.run.smithi007.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.1 MB/s | 1.0 MB 00:00 2024-01-21T03:09:05.584 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:05.584 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2024-01-21T03:09:05.585 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:09:05.585 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 6/6 2024-01-21T03:09:05.585 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/6 2024-01-21T03:09:05.586 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-21T03:09:05.586 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-21T03:09:05.586 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-21T03:09:05.586 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-21T03:09:05.600 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:09:05.600 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:09:05.711 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:09:05.711 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:09:05.895 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-21T03:09:05.895 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:05.895 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:09:05.895 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:09:05.895 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-21T03:09:05.895 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-21T03:09:05.895 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-21T03:09:05.895 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-21T03:09:05.895 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-21T03:09:05.895 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:05.895 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:06.133 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-fuse 2024-01-21T03:09:06.245 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:09:06.419 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-21T03:09:06.485 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-21T03:09:06.549 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-21T03:09:06.585 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-21T03:09:06.627 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-21T03:09:06.699 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:40 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:06.726 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-21T03:09:06.842 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-21T03:09:06.852 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:06.853 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:06.853 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-01-21T03:09:06.853 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:06.853 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:09:06.853 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 886 k 2024-01-21T03:09:06.853 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-01-21T03:09:06.853 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-21T03:09:06.854 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:06.854 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:09:06.854 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:06.854 INFO:teuthology.orchestra.run.smithi177.stdout:Install 2 Packages 2024-01-21T03:09:06.854 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:06.854 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 964 k 2024-01-21T03:09:06.854 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.8 M 2024-01-21T03:09:06.854 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:09:06.895 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-21T03:09:06.996 INFO:teuthology.orchestra.run.smithi177.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 556 kB/s | 79 kB 00:00 2024-01-21T03:09:07.321 INFO:teuthology.orchestra.run.smithi177.stdout:(2/2): ceph-fuse-19.0.0-766.g0edf41a6.el9.x86_6 1.9 MB/s | 886 kB 00:00 2024-01-21T03:09:07.322 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:07.323 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.0 MB/s | 964 kB 00:00 2024-01-21T03:09:07.323 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:09:07.342 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:09:07.343 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:09:07.417 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:09:07.417 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:09:07.614 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:09:07.659 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-21T03:09:07.708 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-21T03:09:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-21T03:09:07.747 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-21T03:09:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-21T03:09:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 2/2 2024-01-21T03:09:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 13/13 2024-01-21T03:09:08.489 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 2/2 2024-01-21T03:09:08.490 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/2 2024-01-21T03:09:08.612 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 13/13 2024-01-21T03:09:08.612 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 1/13 2024-01-21T03:09:08.612 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-21T03:09:08.613 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-21T03:09:08.613 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-21T03:09:08.613 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-21T03:09:08.613 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-21T03:09:08.613 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-21T03:09:08.613 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-21T03:09:08.613 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-21T03:09:08.613 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-21T03:09:08.613 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-21T03:09:08.613 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-21T03:09:08.814 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-21T03:09:08.815 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:08.815 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:09:08.815 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-21T03:09:08.815 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:08.815 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:08.985 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-21T03:09:08.985 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-21T03:09:08.986 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-21T03:09:08.987 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:08.987 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:09.066 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-volume 2024-01-21T03:09:09.225 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T03:09:09.633 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:43 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:09.787 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:09.788 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:09.788 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-01-21T03:09:09.788 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:09.788 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:09:09.788 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 262 k 2024-01-21T03:09:09.788 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:09.788 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:09:09.788 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:09.788 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:09:09.788 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:09.789 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 262 k 2024-01-21T03:09:09.789 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.2 M 2024-01-21T03:09:09.789 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:09:09.801 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:43 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:09.960 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:09.960 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-01-21T03:09:09.960 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 133 k 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2024-01-21T03:09:09.961 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:09.962 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 6.8 M 2024-01-21T03:09:09.962 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 33 M 2024-01-21T03:09:09.962 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:09:10.134 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-volume-19.0.0-766.g0edf41a6.el9.noarch.rpm 761 kB/s | 262 kB 00:00 2024-01-21T03:09:10.134 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:10.134 INFO:teuthology.orchestra.run.smithi177.stdout:Total 758 kB/s | 262 kB 00:00 2024-01-21T03:09:10.134 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:09:10.142 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:09:10.142 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:09:10.179 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:09:10.179 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:09:10.395 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:09:10.468 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:09:10.469 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2024-01-21T03:09:10.484 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:09:10.484 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:09:10.484 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-21T03:09:10.484 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:10.627 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-01-21T03:09:10.744 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): ceph-mgr-cephadm-19.0.0-766.g0edf41a6.el 303 kB/s | 133 kB 00:00 2024-01-21T03:09:11.169 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 111 kB/s | 58 kB 00:00 2024-01-21T03:09:11.236 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 109 kB/s | 51 kB 00:00 2024-01-21T03:09:11.278 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 678 kB/s | 548 kB 00:00 2024-01-21T03:09:11.282 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:11.282 INFO:teuthology.orchestra.run.smithi007.stdout:Total 5.1 MB/s | 6.8 MB 00:01 2024-01-21T03:09:11.283 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:09:11.292 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:09:11.292 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:09:11.386 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:09:11.388 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:09:11.468 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:09:11.468 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:11.468 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:09:11.468 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:09:11.468 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:11.468 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:11.687 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:09:11.768 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados-devel 2024-01-21T03:09:11.849 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-21T03:09:11.906 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-21T03:09:12.348 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:46 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:12.458 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-21T03:09:12.498 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:12.499 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:12.499 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:09:12.499 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:12.499 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:09:12.499 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 127 k 2024-01-21T03:09:12.500 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:12.500 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:09:12.500 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:12.500 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:09:12.500 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:12.500 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 127 k 2024-01-21T03:09:12.500 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 456 k 2024-01-21T03:09:12.500 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:09:12.543 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-21T03:09:12.603 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-21T03:09:12.626 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 6/6 2024-01-21T03:09:12.729 INFO:teuthology.orchestra.run.smithi177.stdout:librados-devel-19.0.0-766.g0edf41a6.el9.x86_64. 553 kB/s | 127 kB 00:00 2024-01-21T03:09:12.730 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:12.730 INFO:teuthology.orchestra.run.smithi177.stdout:Total 550 kB/s | 127 kB 00:00 2024-01-21T03:09:12.730 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:09:12.736 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:09:12.736 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:09:12.789 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:09:12.790 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:09:12.930 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:09:13.027 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:13.434 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 6/6 2024-01-21T03:09:13.434 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/6 2024-01-21T03:09:13.434 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-21T03:09:13.435 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-21T03:09:13.435 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-21T03:09:13.435 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-21T03:09:13.486 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:13.751 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-21T03:09:13.751 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:13.751 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:09:13.751 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:09:13.751 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-21T03:09:13.751 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-21T03:09:13.751 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-21T03:09:13.751 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-21T03:09:13.751 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-21T03:09:13.751 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:13.751 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:13.788 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:13.788 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:13.788 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:09:13.788 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:09:13.788 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:13.788 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:13.988 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse 2024-01-21T03:09:14.090 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs2 2024-01-21T03:09:14.563 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:48 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:14.662 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:48 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:14.721 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:14.722 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:14.722 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-01-21T03:09:14.722 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:14.722 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:09:14.722 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 886 k 2024-01-21T03:09:14.722 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-01-21T03:09:14.722 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-21T03:09:14.722 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:14.722 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:09:14.722 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:14.722 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-01-21T03:09:14.723 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:14.723 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 964 k 2024-01-21T03:09:14.723 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.8 M 2024-01-21T03:09:14.723 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:09:14.768 INFO:teuthology.orchestra.run.smithi177.stdout:Package libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:14.818 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:14.819 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-01-21T03:09:14.819 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:14.847 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 638 kB/s | 79 kB 00:00 2024-01-21T03:09:14.890 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs-devel 2024-01-21T03:09:15.072 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-fuse-19.0.0-766.g0edf41a6.el9.x86_6 2.5 MB/s | 886 kB 00:00 2024-01-21T03:09:15.073 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:15.073 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.7 MB/s | 964 kB 00:00 2024-01-21T03:09:15.073 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:09:15.095 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:09:15.095 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:09:15.173 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:09:15.173 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:09:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:09:15.453 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:49 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-21T03:09:15.582 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 2/2 2024-01-21T03:09:15.604 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:15.605 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:09:15.605 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:15.605 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:09:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 31 k 2024-01-21T03:09:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:15.605 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:09:15.605 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:15.606 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:09:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:15.606 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 31 k 2024-01-21T03:09:15.606 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 140 k 2024-01-21T03:09:15.606 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:09:15.783 INFO:teuthology.orchestra.run.smithi177.stdout:libcephfs-devel-19.0.0-766.g0edf41a6.el9.x86_64 176 kB/s | 31 kB 00:00 2024-01-21T03:09:15.784 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:15.784 INFO:teuthology.orchestra.run.smithi177.stdout:Total 175 kB/s | 31 kB 00:00 2024-01-21T03:09:15.784 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:09:15.787 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:09:15.788 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:09:15.818 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:09:15.818 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:09:15.909 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:09:16.022 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:16.423 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 2/2 2024-01-21T03:09:16.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/2 2024-01-21T03:09:16.692 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:16.692 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:16.692 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:09:16.692 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:09:16.692 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:16.693 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:16.741 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-21T03:09:16.741 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:16.741 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:09:16.741 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-21T03:09:16.741 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:16.741 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:16.867 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados2 2024-01-21T03:09:17.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-volume 2024-01-21T03:09:17.428 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:51 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:17.529 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:17.579 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:17.580 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-01-21T03:09:17.580 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:17.590 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:51 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:17.652 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librbd1 2024-01-21T03:09:17.747 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:17.747 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:17.747 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-01-21T03:09:17.748 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:17.748 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:09:17.748 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 262 k 2024-01-21T03:09:17.748 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:17.748 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:09:17.748 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:17.748 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:09:17.748 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:17.749 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 262 k 2024-01-21T03:09:17.749 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.2 M 2024-01-21T03:09:17.749 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:09:18.102 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-volume-19.0.0-766.g0edf41a6.el9.noarch.rpm 743 kB/s | 262 kB 00:00 2024-01-21T03:09:18.103 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:18.103 INFO:teuthology.orchestra.run.smithi007.stdout:Total 740 kB/s | 262 kB 00:00 2024-01-21T03:09:18.103 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:09:18.111 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:09:18.111 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:09:18.145 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:09:18.146 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:09:18.215 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:52 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:18.315 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:18.365 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:18.366 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-01-21T03:09:18.366 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:18.371 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:09:18.436 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rados 2024-01-21T03:09:18.445 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:09:18.461 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:09:18.461 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:09:18.461 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-21T03:09:18.461 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:18.994 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:53 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:19.095 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:19.145 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:19.146 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-01-21T03:09:19.147 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:19.219 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rgw 2024-01-21T03:09:19.555 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T03:09:19.555 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:19.555 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:09:19.555 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T03:09:19.555 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:19.555 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:19.779 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:53 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:19.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel 2024-01-21T03:09:19.881 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:19.933 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:19.934 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-01-21T03:09:19.934 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:20.005 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-cephfs 2024-01-21T03:09:20.446 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:54 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:20.573 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:54 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:20.603 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:20.604 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:20.604 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:09:20.604 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:20.604 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:09:20.604 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 127 k 2024-01-21T03:09:20.604 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:20.604 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:09:20.604 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:20.604 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:09:20.604 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:20.605 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 127 k 2024-01-21T03:09:20.605 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 456 k 2024-01-21T03:09:20.605 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:09:20.674 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:20.725 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:20.726 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-01-21T03:09:20.726 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:20.794 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rbd 2024-01-21T03:09:21.015 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-19.0.0-766.g0edf41a6.el9.x86_64. 309 kB/s | 127 kB 00:00 2024-01-21T03:09:21.016 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:21.016 INFO:teuthology.orchestra.run.smithi007.stdout:Total 308 kB/s | 127 kB 00:00 2024-01-21T03:09:21.016 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:09:21.022 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:09:21.022 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:09:21.079 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:09:21.080 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:09:21.229 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:09:21.324 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:21.352 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:55 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:21.453 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:21.503 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:21.504 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-01-21T03:09:21.504 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:21.576 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-fuse 2024-01-21T03:09:21.798 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:22.139 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:56 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:22.147 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:22.147 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:22.147 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:09:22.147 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:09:22.147 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:22.147 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-01-21T03:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 85 k 2024-01-21T03:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 85 k 2024-01-21T03:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 231 k 2024-01-21T03:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:09:22.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2 2024-01-21T03:09:22.761 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-fuse-19.0.0-766.g0edf41a6.el9.x86_64.rpm 182 kB/s | 85 kB 00:00 2024-01-21T03:09:22.762 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:22.762 INFO:teuthology.orchestra.run.smithi177.stdout:Total 181 kB/s | 85 kB 00:00 2024-01-21T03:09:22.762 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:09:22.769 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:09:22.769 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:09:22.822 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:09:22.823 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:09:22.957 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:09:22.989 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:56 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:23.058 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:23.093 INFO:teuthology.orchestra.run.smithi007.stdout:Package libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:23.147 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:23.148 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-01-21T03:09:23.148 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:23.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs-devel 2024-01-21T03:09:23.481 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:23.788 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:23.788 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:23.788 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:09:23.788 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:09:23.788 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:23.788 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:23.800 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:57 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:23.958 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:23.959 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:23.959 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:09:23.959 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:23.959 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:09:23.959 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 31 k 2024-01-21T03:09:23.959 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:23.959 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:09:23.959 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:23.959 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:09:23.960 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:23.960 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 31 k 2024-01-21T03:09:23.960 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 140 k 2024-01-21T03:09:23.960 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:09:23.994 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-mirror 2024-01-21T03:09:24.119 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs-devel-19.0.0-766.g0edf41a6.el9.x86_64 196 kB/s | 31 kB 00:00 2024-01-21T03:09:24.120 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:24.120 INFO:teuthology.orchestra.run.smithi007.stdout:Total 194 kB/s | 31 kB 00:00 2024-01-21T03:09:24.120 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:09:24.123 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:09:24.123 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:09:24.155 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:09:24.155 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:09:24.243 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:09:24.361 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:24.574 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:58 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:24.732 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:24.733 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:24.733 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:09:24.733 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:24.733 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:09:24.733 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 3.1 M 2024-01-21T03:09:24.733 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:24.733 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:09:24.733 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:24.733 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:09:24.733 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:24.734 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 3.1 M 2024-01-21T03:09:24.734 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 13 M 2024-01-21T03:09:24.734 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:09:24.763 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:25.038 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:25.038 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:25.038 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:09:25.038 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:09:25.038 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:25.038 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:25.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2 2024-01-21T03:09:25.788 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:59 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:25.847 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-mirror-19.0.0-766.g0edf41a6.el9.x86_64.rpm 2.8 MB/s | 3.1 MB 00:01 2024-01-21T03:09:25.847 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:25.848 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.8 MB/s | 3.1 MB 00:01 2024-01-21T03:09:25.848 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:09:25.858 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:09:25.858 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:09:25.891 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:25.934 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:09:25.934 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:09:25.945 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:25.946 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-01-21T03:09:25.947 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:26.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1 2024-01-21T03:09:26.364 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:26.480 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:26.480 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:09:26.480 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-21T03:09:26.481 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-21T03:09:26.481 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-21T03:09:26.481 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:26.598 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:00 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:26.701 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:26.756 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:26.757 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-01-21T03:09:26.757 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:26.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados 2024-01-21T03:09:27.395 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:01 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:27.499 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:27.553 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:27.554 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-01-21T03:09:27.555 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:27.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw 2024-01-21T03:09:27.642 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:27.642 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:27.642 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:09:27.642 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:09:27.642 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:27.642 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:27.894 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-nbd 2024-01-21T03:09:28.195 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:02 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:28.300 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:28.354 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:28.355 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-01-21T03:09:28.355 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:28.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs 2024-01-21T03:09:28.471 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:02 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:28.622 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:28.622 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:28.622 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-01-21T03:09:28.623 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:28.623 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:09:28.623 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 170 k 2024-01-21T03:09:28.623 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:28.623 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:09:28.623 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:28.623 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:09:28.623 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:28.623 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 170 k 2024-01-21T03:09:28.624 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 490 k 2024-01-21T03:09:28.624 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:09:29.005 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:03 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:29.109 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:29.163 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:29.164 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-01-21T03:09:29.164 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:29.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd 2024-01-21T03:09:29.466 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-nbd-19.0.0-766.g0edf41a6.el9.x86_64.rpm 202 kB/s | 170 kB 00:00 2024-01-21T03:09:29.466 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:29.467 INFO:teuthology.orchestra.run.smithi177.stdout:Total 202 kB/s | 170 kB 00:00 2024-01-21T03:09:29.467 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:09:29.476 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:09:29.476 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:09:29.538 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:09:29.539 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:09:29.690 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:09:29.798 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:29.814 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:03 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:29.921 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:09:29.976 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:29.977 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-01-21T03:09:29.977 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:30.064 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse 2024-01-21T03:09:30.257 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:30.565 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:30.565 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:30.565 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:09:30.565 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:09:30.565 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:30.566 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:09:30.644 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:04 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:30.799 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base-debuginfo 2024-01-21T03:09:30.812 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:30.812 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:30.813 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-01-21T03:09:30.813 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:30.813 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:09:30.813 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 85 k 2024-01-21T03:09:30.813 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:30.813 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:09:30.813 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:30.813 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:09:30.813 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:30.814 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 85 k 2024-01-21T03:09:30.814 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 231 k 2024-01-21T03:09:30.814 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:09:31.298 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-19.0.0-766.g0edf41a6.el9.x86_64.rpm 176 kB/s | 85 kB 00:00 2024-01-21T03:09:31.298 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:31.298 INFO:teuthology.orchestra.run.smithi007.stdout:Total 176 kB/s | 85 kB 00:00 2024-01-21T03:09:31.299 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:09:31.306 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:09:31.306 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:09:31.363 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:09:31.363 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:09:31.371 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:05 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:31.505 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:09:31.530 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:09:31.530 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:31.530 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:09:31.530 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:31.530 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:09:31.530 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 67 M 2024-01-21T03:09:31.530 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-01-21T03:09:31.531 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 22 M 2024-01-21T03:09:31.531 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:09:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:09:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:Install 2 Packages 2024-01-21T03:09:31.531 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:09:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 89 M 2024-01-21T03:09:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 490 M 2024-01-21T03:09:31.532 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:09:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:32.338 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:32.338 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:32.338 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:09:32.338 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:09:32.338 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:32.338 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:32.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror 2024-01-21T03:09:33.116 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:07 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:33.276 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:33.277 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:33.277 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:09:33.277 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:33.277 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:09:33.277 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 3.1 M 2024-01-21T03:09:33.277 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:33.277 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:09:33.277 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:33.277 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:09:33.277 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:33.277 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 3.1 M 2024-01-21T03:09:33.277 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 13 M 2024-01-21T03:09:33.277 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:09:36.650 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-19.0.0-766.g0edf41a6.el9.x86_64.rpm 938 kB/s | 3.1 MB 00:03 2024-01-21T03:09:36.651 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:36.651 INFO:teuthology.orchestra.run.smithi007.stdout:Total 937 kB/s | 3.1 MB 00:03 2024-01-21T03:09:36.652 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:09:36.664 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:09:36.664 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:09:36.743 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:09:36.743 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:09:37.186 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:09:37.268 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:37.292 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T03:09:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-21T03:09:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-21T03:09:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-21T03:09:37.293 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:38.344 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:38.345 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:38.345 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:09:38.345 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:09:38.345 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:38.345 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:38.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd 2024-01-21T03:09:39.176 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:13 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:39.310 INFO:teuthology.orchestra.run.smithi177.stdout:(1/2): ceph-debuginfo-19.0.0-766.g0edf41a6.el9. 2.9 MB/s | 22 MB 00:07 2024-01-21T03:09:39.334 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:39.335 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:39.335 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-01-21T03:09:39.335 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:39.335 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:09:39.335 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 170 k 2024-01-21T03:09:39.336 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:39.336 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:09:39.336 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:39.336 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:09:39.336 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:39.336 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 170 k 2024-01-21T03:09:39.336 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 490 k 2024-01-21T03:09:39.336 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:09:40.023 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-19.0.0-766.g0edf41a6.el9.x86_64.rpm 248 kB/s | 170 kB 00:00 2024-01-21T03:09:40.023 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:40.024 INFO:teuthology.orchestra.run.smithi007.stdout:Total 248 kB/s | 170 kB 00:00 2024-01-21T03:09:40.024 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:09:40.033 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:09:40.033 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:09:40.097 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:09:40.098 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:09:40.251 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:09:40.348 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:40.813 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:41.153 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:09:41.154 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:41.154 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:09:41.154 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:09:41.154 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:41.154 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:09:41.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base-debuginfo 2024-01-21T03:09:41.962 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:15 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:09:42.119 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:09:42.119 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:09:42.119 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:42.119 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:09:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 67 M 2024-01-21T03:09:42.120 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-01-21T03:09:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 22 M 2024-01-21T03:09:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:42.120 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:09:42.120 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:09:42.120 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-01-21T03:09:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:09:42.121 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 89 M 2024-01-21T03:09:42.121 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 490 M 2024-01-21T03:09:42.121 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:09:47.517 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): ceph-debuginfo-19.0.0-766.g0edf41a6.el9. 4.1 MB/s | 22 MB 00:05 2024-01-21T03:09:49.284 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-base-debuginfo-19.0.0-766.g0edf41a6 9.3 MB/s | 67 MB 00:07 2024-01-21T03:09:49.286 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:49.286 INFO:teuthology.orchestra.run.smithi007.stdout:Total 12 MB/s | 89 MB 00:07 2024-01-21T03:09:49.286 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:09:49.291 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:09:49.291 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:09:49.891 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:09:49.891 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:09:53.148 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:09:56.390 INFO:teuthology.orchestra.run.smithi177.stdout:(2/2): ceph-base-debuginfo-19.0.0-766.g0edf41a6 2.7 MB/s | 67 MB 00:24 2024-01-21T03:09:56.392 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:09:56.392 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.6 MB/s | 89 MB 00:24 2024-01-21T03:09:56.393 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:09:56.397 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:09:56.397 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:09:56.975 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:09:56.975 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:10:00.058 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:10:00.528 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/2 2024-01-21T03:10:00.604 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 2/2 2024-01-21T03:10:04.124 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 2/2 2024-01-21T03:10:04.124 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/2 2024-01-21T03:10:04.408 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2/2 2024-01-21T03:10:04.408 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:10:04.408 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:10:04.408 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:10:04.408 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:10:04.409 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:10:04.409 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:10:04.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-common-debuginfo 2024-01-21T03:10:05.204 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:39 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:10:05.363 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:10:05.363 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:10:05.363 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:10:05.364 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:10:05.364 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:10:05.364 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 411 M 2024-01-21T03:10:05.364 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:10:05.364 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:10:05.364 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:10:05.364 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:10:05.364 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:10:05.364 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 411 M 2024-01-21T03:10:05.364 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.7 G 2024-01-21T03:10:05.364 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:10:07.068 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/2 2024-01-21T03:10:07.224 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 2/2 2024-01-21T03:10:10.494 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 2/2 2024-01-21T03:10:10.494 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/2 2024-01-21T03:10:10.784 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2/2 2024-01-21T03:10:10.784 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:10:10.784 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:10:10.784 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:10:10.785 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:10:10.785 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:10:10.785 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:10:11.005 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-common-debuginfo 2024-01-21T03:10:11.580 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:45 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:10:11.742 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:10:11.743 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:10:11.743 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:10:11.743 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:10:11.744 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:10:11.744 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 411 M 2024-01-21T03:10:11.744 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:10:11.744 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:10:11.744 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:10:11.744 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:10:11.744 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:10:11.744 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 411 M 2024-01-21T03:10:11.744 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.7 G 2024-01-21T03:10:11.744 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:11:21.751 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-common-debuginfo-19.0.0-766.g0edf41a6.el9. 5.9 MB/s | 411 MB 01:10 2024-01-21T03:11:21.751 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:11:21.752 INFO:teuthology.orchestra.run.smithi177.stdout:Total 5.9 MB/s | 411 MB 01:10 2024-01-21T03:11:21.752 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:11:21.755 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:11:21.755 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:11:24.197 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:11:24.197 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:11:26.268 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-common-debuginfo-19.0.0-766.g0edf41a6.el9. 5.1 MB/s | 411 MB 01:20 2024-01-21T03:11:26.269 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:11:26.269 INFO:teuthology.orchestra.run.smithi007.stdout:Total 5.1 MB/s | 411 MB 01:20 2024-01-21T03:11:26.270 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:11:26.274 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:11:26.274 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:11:28.835 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:11:28.835 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:12:05.849 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:12:05.951 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:12:13.527 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:12:13.631 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:12:14.699 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:12:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:12:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:12:14.971 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:12:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:12:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:12:14.971 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:12:15.223 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-01-21T03:12:15.789 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:04:49 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:12:15.940 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:12:15.941 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-01-21T03:12:15.941 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:12:15.941 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-01-21T03:12:15.941 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:12:15.941 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.4 M 2024-01-21T03:12:15.941 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:12:15.941 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:12:15.941 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-01-21T03:12:15.941 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:12:15.941 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:12:15.941 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 1.4 M 2024-01-21T03:12:15.941 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 6.4 M 2024-01-21T03:12:15.941 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:12:16.896 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-76 1.5 MB/s | 1.4 MB 00:00 2024-01-21T03:12:16.897 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:12:16.898 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.5 MB/s | 1.4 MB 00:00 2024-01-21T03:12:16.898 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:12:16.901 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:12:16.901 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:12:16.946 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:12:16.946 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:12:17.212 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:12:17.327 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g 1/1 2024-01-21T03:12:17.738 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g 1/1 2024-01-21T03:12:18.016 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g 1/1 2024-01-21T03:12:18.017 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:12:18.017 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:12:18.017 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:12:18.017 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:12:18.017 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:12:18.206 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw-debuginfo 2024-01-21T03:12:18.616 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:12:18.763 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:04:52 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:12:18.892 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:12:18.892 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:12:18.892 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:12:18.892 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:12:18.892 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:12:18.892 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:12:18.912 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:12:18.913 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:12:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:12:18.913 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:12:18.913 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:12:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 170 M 2024-01-21T03:12:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:12:18.913 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:12:18.913 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:12:18.913 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:12:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:12:18.914 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 170 M 2024-01-21T03:12:18.914 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 721 M 2024-01-21T03:12:18.914 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:12:19.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-01-21T03:12:19.707 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:53 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:12:19.867 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:12:19.868 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-01-21T03:12:19.868 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:12:19.868 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-01-21T03:12:19.868 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:12:19.868 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.4 M 2024-01-21T03:12:19.868 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:12:19.868 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:12:19.868 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-01-21T03:12:19.868 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:12:19.868 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:12:19.868 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 1.4 M 2024-01-21T03:12:19.868 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 6.4 M 2024-01-21T03:12:19.868 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:12:20.358 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-76 2.9 MB/s | 1.4 MB 00:00 2024-01-21T03:12:20.359 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:12:20.359 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.9 MB/s | 1.4 MB 00:00 2024-01-21T03:12:20.359 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:12:20.363 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:12:20.363 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:12:20.406 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:12:20.407 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:12:20.700 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:12:20.817 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g 1/1 2024-01-21T03:12:21.249 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g 1/1 2024-01-21T03:12:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g 1/1 2024-01-21T03:12:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:12:21.532 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:12:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:12:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:12:21.532 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:12:21.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw-debuginfo 2024-01-21T03:12:22.294 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:56 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:12:22.451 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:12:22.452 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:12:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:12:22.452 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:12:22.452 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:12:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 170 M 2024-01-21T03:12:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:12:22.452 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:12:22.452 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:12:22.452 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:12:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:12:22.452 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 170 M 2024-01-21T03:12:22.453 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 721 M 2024-01-21T03:12:22.453 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:12:32.156 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-radosgw-debuginfo-19.0.0-766.g0edf41a6.el9 13 MB/s | 170 MB 00:13 2024-01-21T03:12:32.156 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:12:32.157 INFO:teuthology.orchestra.run.smithi177.stdout:Total 13 MB/s | 170 MB 00:13 2024-01-21T03:12:32.157 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:12:32.161 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:12:32.161 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:12:33.188 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:12:33.189 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:12:40.367 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-radosgw-debuginfo-19.0.0-766.g0edf41a6.el9 9.5 MB/s | 170 MB 00:17 2024-01-21T03:12:40.368 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:12:40.368 INFO:teuthology.orchestra.run.smithi007.stdout:Total 9.5 MB/s | 170 MB 00:17 2024-01-21T03:12:40.369 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:12:40.373 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:12:40.373 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:12:41.451 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:12:41.452 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:12:50.650 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:12:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x8 1/1 2024-01-21T03:12:55.168 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x8 1/1 2024-01-21T03:12:55.444 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x8 1/1 2024-01-21T03:12:55.444 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:12:55.444 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:12:55.444 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:12:55.444 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:12:55.444 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:12:55.645 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-test-debuginfo 2024-01-21T03:12:56.208 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:05:30 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:12:56.361 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:12:56.362 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:12:56.362 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:12:56.362 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:12:56.362 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:12:56.362 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 664 M 2024-01-21T03:12:56.362 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:12:56.362 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:12:56.362 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:12:56.362 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:12:56.362 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:12:56.362 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 664 M 2024-01-21T03:12:56.362 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 3.0 G 2024-01-21T03:12:56.362 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:12:59.951 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:13:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x8 1/1 2024-01-21T03:13:04.699 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x8 1/1 2024-01-21T03:13:04.983 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x8 1/1 2024-01-21T03:13:04.983 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:13:04.983 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:13:04.983 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:13:04.983 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:13:04.983 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:13:05.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test-debuginfo 2024-01-21T03:13:05.749 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:05:39 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:13:05.908 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:13:05.910 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:13:05.910 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:13:05.910 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:13:05.910 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:13:05.910 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 664 M 2024-01-21T03:13:05.910 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:13:05.910 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:13:05.910 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:13:05.910 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:13:05.911 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:13:05.911 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 664 M 2024-01-21T03:13:05.911 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 3.0 G 2024-01-21T03:13:05.911 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:13:38.974 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-test-debuginfo-19.0.0-766.g0edf41a6.el9.x8 16 MB/s | 664 MB 00:42 2024-01-21T03:13:38.975 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:13:38.975 INFO:teuthology.orchestra.run.smithi177.stdout:Total 16 MB/s | 664 MB 00:42 2024-01-21T03:13:38.975 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:13:38.981 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:13:38.981 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:13:42.917 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:13:42.917 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:14:02.110 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-test-debuginfo-19.0.0-766.g0edf41a6.el9.x8 12 MB/s | 664 MB 00:56 2024-01-21T03:14:02.110 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:14:02.111 INFO:teuthology.orchestra.run.smithi007.stdout:Total 12 MB/s | 664 MB 00:56 2024-01-21T03:14:02.111 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:14:02.117 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:14:02.117 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:14:06.220 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:14:06.220 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:14:53.678 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:14:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:15:02.897 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:15:03.238 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:15:03.238 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:03.238 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:15:03.238 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:15:03.238 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:03.238 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:15:03.519 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base-debuginfo 2024-01-21T03:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:07:38 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:15:04.184 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:15:04.236 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:15:04.237 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-01-21T03:15:04.237 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:15:04.315 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-debuginfo 2024-01-21T03:15:04.875 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:07:38 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:15:05.029 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:15:05.030 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:15:05.030 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:15:05.030 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:15:05.030 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:15:05.031 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 20 M 2024-01-21T03:15:05.031 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:05.031 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:15:05.031 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:15:05.031 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:15:05.031 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:05.031 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 20 M 2024-01-21T03:15:05.031 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 102 M 2024-01-21T03:15:05.031 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:15:06.298 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-mgr-debuginfo-19.0.0-766.g0edf41a6.el9.x86 16 MB/s | 20 MB 00:01 2024-01-21T03:15:06.299 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:15:06.299 INFO:teuthology.orchestra.run.smithi177.stdout:Total 15 MB/s | 20 MB 00:01 2024-01-21T03:15:06.300 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:15:06.303 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:15:06.303 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:15:06.453 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:15:06.454 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:15:08.735 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:15:08.829 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:09.803 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:10.068 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:10.068 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:10.068 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:15:10.068 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:15:10.068 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:10.069 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:15:10.252 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mds-debuginfo 2024-01-21T03:15:10.817 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:07:44 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:15:10.968 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 48 M 2024-01-21T03:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 48 M 2024-01-21T03:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 173 M 2024-01-21T03:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:15:18.425 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-mds-debuginfo-19.0.0-766.g0edf41a6.el9.x86 6.4 MB/s | 48 MB 00:07 2024-01-21T03:15:18.425 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:15:18.426 INFO:teuthology.orchestra.run.smithi177.stdout:Total 6.4 MB/s | 48 MB 00:07 2024-01-21T03:15:18.426 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:15:18.430 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:15:18.430 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:15:18.741 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:15:18.742 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:15:20.439 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:15:21.062 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:15:23.469 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:15:23.563 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:24.926 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:25.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:25.204 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:25.204 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:15:25.204 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:15:25.204 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:25.204 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:15:25.378 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mon-debuginfo 2024-01-21T03:15:25.937 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:07:59 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:15:26.087 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:15:26.088 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:15:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:15:26.088 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:15:26.088 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:15:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 99 M 2024-01-21T03:15:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:26.088 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:15:26.088 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:15:26.088 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:15:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:26.089 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 99 M 2024-01-21T03:15:26.089 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 449 M 2024-01-21T03:15:26.089 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:15:29.651 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:15:29.923 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:15:29.923 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:15:29.923 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:15:29.924 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:15:29.924 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:15:29.924 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:15:30.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base-debuginfo 2024-01-21T03:15:30.576 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-mon-debuginfo-19.0.0-766.g0edf41a6.el9.x86 22 MB/s | 99 MB 00:04 2024-01-21T03:15:30.577 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:15:30.578 INFO:teuthology.orchestra.run.smithi177.stdout:Total 22 MB/s | 99 MB 00:04 2024-01-21T03:15:30.578 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:15:30.582 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:15:30.582 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:15:30.779 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:04 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:15:30.883 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T03:15:30.938 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:15:30.939 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-01-21T03:15:30.939 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:15:31.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-debuginfo 2024-01-21T03:15:31.194 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:15:31.194 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:15:31.579 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:05 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:15:31.737 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:15:31.738 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:15:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:15:31.738 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:15:31.738 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:15:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 20 M 2024-01-21T03:15:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:15:31.738 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:15:31.738 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:15:31.738 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:15:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:15:31.739 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 20 M 2024-01-21T03:15:31.739 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 102 M 2024-01-21T03:15:31.739 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:15:32.827 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-mgr-debuginfo-19.0.0-766.g0edf41a6.el9.x86 18 MB/s | 20 MB 00:01 2024-01-21T03:15:32.828 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:15:32.829 INFO:teuthology.orchestra.run.smithi007.stdout:Total 18 MB/s | 20 MB 00:01 2024-01-21T03:15:32.829 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:15:32.833 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:15:32.833 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:15:32.989 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:15:32.989 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:15:35.419 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:15:35.528 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:36.533 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:36.821 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:36.821 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:15:36.822 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:15:36.822 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:15:36.822 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:15:36.822 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:15:36.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mds-debuginfo 2024-01-21T03:15:37.568 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:11 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:15:37.727 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:15:37.727 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:15:37.727 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:15:37.727 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:15:37.727 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:15:37.727 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 48 M 2024-01-21T03:15:37.727 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:15:37.727 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:15:37.727 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:15:37.727 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:15:37.727 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:15:37.728 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 48 M 2024-01-21T03:15:37.728 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 173 M 2024-01-21T03:15:37.728 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:15:40.014 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-mds-debuginfo-19.0.0-766.g0edf41a6.el9.x86 21 MB/s | 48 MB 00:02 2024-01-21T03:15:40.014 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout:Total 21 MB/s | 48 MB 00:02 2024-01-21T03:15:40.015 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:15:40.019 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:15:40.019 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:15:40.346 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:15:40.346 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:15:41.518 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:15:41.623 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:44.559 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:44.828 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:44.828 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:44.828 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:15:44.828 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:15:44.828 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:44.828 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:15:45.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-osd-debuginfo 2024-01-21T03:15:45.364 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:15:45.455 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:45.607 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:19 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:15:45.760 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:15:45.761 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:15:45.761 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:15:45.761 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:15:45.761 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:15:45.762 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 286 M 2024-01-21T03:15:45.762 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:45.762 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:15:45.762 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:15:45.762 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:15:45.762 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:15:45.763 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 286 M 2024-01-21T03:15:45.763 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.2 G 2024-01-21T03:15:45.763 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:15:46.845 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:47.128 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:15:47.128 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:15:47.128 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:15:47.128 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:15:47.128 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:15:47.128 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:15:47.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mon-debuginfo 2024-01-21T03:15:47.853 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:21 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:15:48.012 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:15:48.013 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:15:48.013 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:15:48.013 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:15:48.013 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:15:48.013 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 99 M 2024-01-21T03:15:48.013 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:15:48.013 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:15:48.013 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:15:48.014 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:15:48.014 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:15:48.014 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 99 M 2024-01-21T03:15:48.014 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 449 M 2024-01-21T03:15:48.014 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:16:01.189 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-mon-debuginfo-19.0.0-766.g0edf41a6.el9.x86 7.5 MB/s | 99 MB 00:13 2024-01-21T03:16:01.190 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:16:01.190 INFO:teuthology.orchestra.run.smithi007.stdout:Total 7.5 MB/s | 99 MB 00:13 2024-01-21T03:16:01.191 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:16:01.195 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:16:01.195 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:16:01.833 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:16:01.833 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:16:01.950 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-osd-debuginfo-19.0.0-766.g0edf41a6.el9.x86 18 MB/s | 286 MB 00:16 2024-01-21T03:16:01.951 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:16:01.951 INFO:teuthology.orchestra.run.smithi177.stdout:Total 18 MB/s | 286 MB 00:16 2024-01-21T03:16:01.951 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:16:01.955 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:16:01.955 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:16:03.660 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:16:03.660 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:16:12.715 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:16:12.831 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:16:15.807 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:16:16.091 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:16:16.092 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:16:16.092 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:16:16.092 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:16:16.092 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:16:16.092 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:16:16.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-osd-debuginfo 2024-01-21T03:16:16.823 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:50 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:16:16.982 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:16:16.983 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:16:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:16:16.983 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:16:16.983 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:16:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 286 M 2024-01-21T03:16:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:16:16.983 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:16:16.983 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:16:16.983 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:16:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:16:16.983 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 286 M 2024-01-21T03:16:16.984 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.2 G 2024-01-21T03:16:16.984 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:16:29.664 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-osd-debuginfo-19.0.0-766.g0edf41a6.el9.x86 23 MB/s | 286 MB 00:12 2024-01-21T03:16:29.665 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:16:29.666 INFO:teuthology.orchestra.run.smithi007.stdout:Total 23 MB/s | 286 MB 00:12 2024-01-21T03:16:29.666 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:16:29.670 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:16:29.670 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:16:31.455 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:16:31.455 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:16:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:16:32.969 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:16:40.250 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:16:40.518 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:16:40.518 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:40.519 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:16:40.519 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:16:40.519 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:40.519 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:16:40.725 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-fuse-debuginfo 2024-01-21T03:16:41.286 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:09:15 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:16:41.439 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:16:41.440 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:16:41.440 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:16:41.440 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:16:41.440 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:16:41.440 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 5.9 M 2024-01-21T03:16:41.441 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:41.441 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:16:41.441 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:16:41.441 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:16:41.441 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:41.441 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 5.9 M 2024-01-21T03:16:41.441 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 28 M 2024-01-21T03:16:41.441 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:16:41.949 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-fuse-debuginfo-19.0.0-766.g0edf41a6.el9.x8 12 MB/s | 5.9 MB 00:00 2024-01-21T03:16:41.950 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:16:41.950 INFO:teuthology.orchestra.run.smithi177.stdout:Total 12 MB/s | 5.9 MB 00:00 2024-01-21T03:16:41.951 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:16:41.954 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:16:41.954 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:16:42.022 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:16:42.023 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:16:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:16:42.855 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:16:43.415 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:16:43.679 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:16:43.679 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:43.679 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:16:43.679 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:16:43.679 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:43.679 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:16:43.841 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados-devel-debuginfo 2024-01-21T03:16:44.402 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:09:18 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:16:44.552 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:16:44.553 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:16:44.553 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:16:44.553 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:16:44.553 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:16:44.553 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 109 k 2024-01-21T03:16:44.553 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:44.553 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:16:44.553 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:16:44.554 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:16:44.554 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:44.554 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 109 k 2024-01-21T03:16:44.554 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 439 k 2024-01-21T03:16:44.554 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:16:44.793 INFO:teuthology.orchestra.run.smithi177.stdout:librados-devel-debuginfo-19.0.0-766.g0edf41a6.e 453 kB/s | 109 kB 00:00 2024-01-21T03:16:44.794 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:16:44.794 INFO:teuthology.orchestra.run.smithi177.stdout:Total 451 kB/s | 109 kB 00:00 2024-01-21T03:16:44.794 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:16:44.798 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:16:44.798 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:16:44.831 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:16:44.831 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:16:44.946 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:16:45.050 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T03:16:45.399 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T03:16:45.675 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T03:16:45.675 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:45.675 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:16:45.675 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:16:45.675 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:45.675 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:16:45.851 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs2-debuginfo 2024-01-21T03:16:46.414 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:09:20 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:16:46.567 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:16:46.568 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:16:46.568 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:16:46.568 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:16:46.568 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:16:46.568 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 5.3 M 2024-01-21T03:16:46.569 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:46.569 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:16:46.569 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:16:46.569 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:16:46.569 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:46.569 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 5.3 M 2024-01-21T03:16:46.569 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 25 M 2024-01-21T03:16:46.569 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:16:47.472 INFO:teuthology.orchestra.run.smithi177.stdout:libcephfs2-debuginfo-19.0.0-766.g0edf41a6.el9.x 5.9 MB/s | 5.3 MB 00:00 2024-01-21T03:16:47.472 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:16:47.473 INFO:teuthology.orchestra.run.smithi177.stdout:Total 5.9 MB/s | 5.3 MB 00:00 2024-01-21T03:16:47.473 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:16:47.476 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:16:47.476 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:16:47.541 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:16:47.541 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:16:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:16:48.311 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T03:16:48.848 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T03:16:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T03:16:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:49.149 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:16:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:16:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:49.149 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:16:49.334 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados2-debuginfo 2024-01-21T03:16:49.892 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:09:23 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:16:50.044 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:16:50.045 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:16:50.045 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:16:50.045 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:16:50.045 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:16:50.045 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 44 M 2024-01-21T03:16:50.045 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:50.045 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:16:50.045 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:16:50.046 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:16:50.046 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:16:50.046 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 44 M 2024-01-21T03:16:50.046 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 252 M 2024-01-21T03:16:50.046 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:16:52.682 INFO:teuthology.orchestra.run.smithi177.stdout:librados2-debuginfo-19.0.0-766.g0edf41a6.el9.x8 17 MB/s | 44 MB 00:02 2024-01-21T03:16:52.683 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:16:52.684 INFO:teuthology.orchestra.run.smithi177.stdout:Total 17 MB/s | 44 MB 00:02 2024-01-21T03:16:52.684 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:16:52.688 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:16:52.688 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:16:52.990 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:16:52.990 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:16:57.997 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:16:58.096 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:17:00.022 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:17:00.296 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:17:00.296 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:00.296 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:17:00.296 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:00.296 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:00.296 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:17:00.504 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librbd1-debuginfo 2024-01-21T03:17:01.065 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:09:35 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:01.216 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:17:01.216 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:01.217 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:17:01.217 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:01.217 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:17:01.217 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 52 M 2024-01-21T03:17:01.217 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:01.217 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:17:01.217 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:01.217 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:17:01.217 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:01.217 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 52 M 2024-01-21T03:17:01.217 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 308 M 2024-01-21T03:17:01.217 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:17:02.499 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:17:02.586 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:03.685 INFO:teuthology.orchestra.run.smithi177.stdout:librbd1-debuginfo-19.0.0-766.g0edf41a6.el9.x86_ 21 MB/s | 52 MB 00:02 2024-01-21T03:17:03.685 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:03.686 INFO:teuthology.orchestra.run.smithi177.stdout:Total 21 MB/s | 52 MB 00:02 2024-01-21T03:17:03.686 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:17:03.690 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:17:03.690 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:17:04.037 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:17:04.037 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:17:10.033 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:10.036 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:17:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:10.305 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:10.306 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:10.306 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:17:10.306 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:10.306 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:10.306 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:17:10.490 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse-debuginfo 2024-01-21T03:17:11.060 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:09:45 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:11.218 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:17:11.219 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:17:11.219 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:11.219 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:17:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 5.9 M 2024-01-21T03:17:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:11.219 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:17:11.219 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:11.219 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:17:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:11.219 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 5.9 M 2024-01-21T03:17:11.219 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 28 M 2024-01-21T03:17:11.219 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:17:12.187 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-fuse-debuginfo-19.0.0-766.g0edf41a6.el9.x8 6.1 MB/s | 5.9 MB 00:00 2024-01-21T03:17:12.188 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:12.188 INFO:teuthology.orchestra.run.smithi007.stdout:Total 6.1 MB/s | 5.9 MB 00:00 2024-01-21T03:17:12.189 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:17:12.192 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:17:12.192 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:17:12.262 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:17:12.263 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:17:12.328 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:12.596 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:12.596 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:12.596 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:17:12.596 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:12.596 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:12.596 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:17:12.848 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-cephfs-debuginfo 2024-01-21T03:17:13.033 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:17:13.130 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:17:13.412 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:09:47 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:13.562 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:17:13.563 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:13.563 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:17:13.563 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:13.563 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:17:13.563 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 615 k 2024-01-21T03:17:13.563 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:13.564 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:17:13.564 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:13.564 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:17:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:13.564 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 615 k 2024-01-21T03:17:13.564 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.1 M 2024-01-21T03:17:13.564 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:17:13.727 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:17:13.845 INFO:teuthology.orchestra.run.smithi177.stdout:python3-cephfs-debuginfo-19.0.0-766.g0edf41a6.e 2.1 MB/s | 615 kB 00:00 2024-01-21T03:17:13.845 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:13.846 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.1 MB/s | 615 kB 00:00 2024-01-21T03:17:13.846 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:17:13.849 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:17:13.849 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:17:13.886 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:17:13.886 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:17:14.013 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:17:14.013 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:14.013 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:17:14.013 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:14.013 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:14.013 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:17:14.049 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:17:14.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel-debuginfo 2024-01-21T03:17:14.177 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T03:17:14.577 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T03:17:14.736 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:09:48 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:14.853 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T03:17:14.853 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:14.853 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:17:14.853 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:14.853 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:14.853 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:17:14.893 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:17:14.893 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:14.893 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:17:14.893 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:14.893 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:17:14.893 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 109 k 2024-01-21T03:17:14.894 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:14.894 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:17:14.894 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:14.894 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:17:14.894 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:14.894 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 109 k 2024-01-21T03:17:14.894 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 439 k 2024-01-21T03:17:14.895 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:17:15.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rados-debuginfo 2024-01-21T03:17:15.230 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-debuginfo-19.0.0-766.g0edf41a6.e 324 kB/s | 109 kB 00:00 2024-01-21T03:17:15.230 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:15.231 INFO:teuthology.orchestra.run.smithi007.stdout:Total 322 kB/s | 109 kB 00:00 2024-01-21T03:17:15.231 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:17:15.234 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:17:15.234 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:17:15.269 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:17:15.270 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:17:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:17:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T03:17:15.600 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:09:49 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:15.751 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:17:15.752 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:15.752 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:17:15.752 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:15.752 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:17:15.752 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 769 k 2024-01-21T03:17:15.752 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:15.752 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:17:15.752 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:15.752 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:17:15.752 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:15.753 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 769 k 2024-01-21T03:17:15.753 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.7 M 2024-01-21T03:17:15.753 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:17:15.878 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T03:17:16.156 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T03:17:16.156 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:16.156 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:17:16.156 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:16.156 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:16.156 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:17:16.181 INFO:teuthology.orchestra.run.smithi177.stdout:python3-rados-debuginfo-19.0.0-766.g0edf41a6.el 1.8 MB/s | 769 kB 00:00 2024-01-21T03:17:16.182 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:16.182 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.7 MB/s | 769 kB 00:00 2024-01-21T03:17:16.182 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:17:16.185 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:17:16.185 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:17:16.223 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:17:16.223 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:17:16.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2-debuginfo 2024-01-21T03:17:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:17:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x 1/1 2024-01-21T03:17:16.887 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x 1/1 2024-01-21T03:17:16.907 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:09:50 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:17.067 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:17:17.068 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:17.068 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:17:17.068 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:17.068 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:17:17.068 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 5.3 M 2024-01-21T03:17:17.068 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:17.068 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:17:17.068 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:17.068 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:17:17.068 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:17.069 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 5.3 M 2024-01-21T03:17:17.069 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 25 M 2024-01-21T03:17:17.069 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:17:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x 1/1 2024-01-21T03:17:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:17.158 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:17:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:17.159 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:17:17.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rbd-debuginfo 2024-01-21T03:17:17.906 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:09:51 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:17.927 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs2-debuginfo-19.0.0-766.g0edf41a6.el9.x 6.2 MB/s | 5.3 MB 00:00 2024-01-21T03:17:17.927 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:17.928 INFO:teuthology.orchestra.run.smithi007.stdout:Total 6.2 MB/s | 5.3 MB 00:00 2024-01-21T03:17:17.928 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:17:17.931 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:17:17.932 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:17:17.999 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:17:17.999 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:17:18.058 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:17:18.059 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:18.059 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:17:18.059 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:18.059 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:17:18.059 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.1 M 2024-01-21T03:17:18.059 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:18.059 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:17:18.060 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:18.060 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:17:18.060 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:18.060 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 1.1 M 2024-01-21T03:17:18.060 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 4.3 M 2024-01-21T03:17:18.060 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:17:18.368 INFO:teuthology.orchestra.run.smithi177.stdout:python3-rbd-debuginfo-19.0.0-766.g0edf41a6.el9. 3.7 MB/s | 1.1 MB 00:00 2024-01-21T03:17:18.368 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:18.369 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.6 MB/s | 1.1 MB 00:00 2024-01-21T03:17:18.369 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:17:18.372 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:17:18.372 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:17:18.412 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:17:18.412 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:17:18.627 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:17:18.700 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:17:18.733 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:17:18.795 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T03:17:19.143 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:17:19.381 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T03:17:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:17:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:19.437 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:17:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:19.437 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:17:19.632 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rgw-debuginfo 2024-01-21T03:17:19.691 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T03:17:19.691 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:19.691 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:17:19.691 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:19.691 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:19.691 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:17:19.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2-debuginfo 2024-01-21T03:17:20.195 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:09:54 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:20.348 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:17:20.349 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:20.349 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:17:20.349 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:20.349 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:17:20.349 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 337 k 2024-01-21T03:17:20.349 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:20.349 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:17:20.349 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:20.349 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:17:20.349 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:20.350 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 337 k 2024-01-21T03:17:20.350 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.0 M 2024-01-21T03:17:20.350 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:17:20.429 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:09:54 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:20.587 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:17:20.588 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:20.588 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:17:20.588 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:20.588 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:17:20.588 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 44 M 2024-01-21T03:17:20.589 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:20.589 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:17:20.589 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:20.589 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:17:20.589 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:20.589 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 44 M 2024-01-21T03:17:20.589 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 252 M 2024-01-21T03:17:20.589 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:17:20.604 INFO:teuthology.orchestra.run.smithi177.stdout:python3-rgw-debuginfo-19.0.0-766.g0edf41a6.el9. 1.3 MB/s | 337 kB 00:00 2024-01-21T03:17:20.604 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:20.605 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.3 MB/s | 337 kB 00:00 2024-01-21T03:17:20.605 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:17:20.608 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:17:20.608 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:17:20.644 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:17:20.644 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:17:20.779 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:17:20.886 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:17:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:17:21.585 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:17:21.585 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:21.585 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:17:21.586 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:21.586 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:21.586 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:17:21.780 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-fuse-debuginfo 2024-01-21T03:17:22.341 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:09:56 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:22.493 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:17:22.494 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:22.494 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:17:22.494 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:22.494 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:17:22.494 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 509 k 2024-01-21T03:17:22.494 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:22.494 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:17:22.494 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:22.494 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:17:22.494 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:22.495 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 509 k 2024-01-21T03:17:22.495 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.5 M 2024-01-21T03:17:22.495 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:17:22.908 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-fuse-debuginfo-19.0.0-766.g0edf41a6.el9.x86 1.2 MB/s | 509 kB 00:00 2024-01-21T03:17:22.908 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:22.908 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.2 MB/s | 509 kB 00:00 2024-01-21T03:17:22.909 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:17:22.912 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:17:22.912 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:17:22.948 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:17:22.949 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:17:23.109 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:17:23.146 INFO:teuthology.orchestra.run.smithi007.stdout:librados2-debuginfo-19.0.0-766.g0edf41a6.el9.x8 17 MB/s | 44 MB 00:02 2024-01-21T03:17:23.148 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:23.148 INFO:teuthology.orchestra.run.smithi007.stdout:Total 17 MB/s | 44 MB 00:02 2024-01-21T03:17:23.148 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:17:23.152 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:17:23.152 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:17:23.205 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:23.468 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:17:23.468 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:17:23.578 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:23.841 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:23.841 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:23.841 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:17:23.841 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:23.841 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:23.841 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:17:24.022 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-mirror-debuginfo 2024-01-21T03:17:24.581 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:09:58 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:24.733 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:17:24.734 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:24.734 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:17:24.734 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:24.734 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:17:24.734 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 52 M 2024-01-21T03:17:24.734 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:24.734 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:17:24.734 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:24.734 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:17:24.734 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:24.734 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 52 M 2024-01-21T03:17:24.734 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 298 M 2024-01-21T03:17:24.734 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:17:27.493 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-mirror-debuginfo-19.0.0-766.g0edf41a6.el9.x 19 MB/s | 52 MB 00:02 2024-01-21T03:17:27.494 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:27.495 INFO:teuthology.orchestra.run.smithi177.stdout:Total 19 MB/s | 52 MB 00:02 2024-01-21T03:17:27.495 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:17:27.499 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:17:27.499 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:17:27.834 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:17:27.834 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:17:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:17:28.906 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:17:30.794 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:17:31.061 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T03:17:31.061 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:31.062 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:17:31.062 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:31.062 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:31.062 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:17:31.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1-debuginfo 2024-01-21T03:17:31.825 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:10:05 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:31.982 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:17:31.982 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:17:31.983 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:31.983 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:17:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 52 M 2024-01-21T03:17:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:31.983 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:17:31.983 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:31.983 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:17:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:31.984 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 52 M 2024-01-21T03:17:31.984 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 308 M 2024-01-21T03:17:31.984 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:17:33.715 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:17:33.816 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T03:17:34.707 INFO:teuthology.orchestra.run.smithi007.stdout:librbd1-debuginfo-19.0.0-766.g0edf41a6.el9.x86_ 19 MB/s | 52 MB 00:02 2024-01-21T03:17:34.708 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:34.708 INFO:teuthology.orchestra.run.smithi007.stdout:Total 19 MB/s | 52 MB 00:02 2024-01-21T03:17:34.708 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:17:34.712 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:17:34.713 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:17:35.074 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:17:35.075 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:17:36.154 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T03:17:36.438 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T03:17:36.438 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:36.439 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:17:36.439 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:36.439 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:36.439 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:17:36.640 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-nbd-debuginfo 2024-01-21T03:17:37.213 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:10:11 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:37.364 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-01-21T03:17:37.365 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:37.365 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-01-21T03:17:37.365 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:37.366 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-01-21T03:17:37.366 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.3 M 2024-01-21T03:17:37.366 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:37.366 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-01-21T03:17:37.366 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-01-21T03:17:37.366 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-01-21T03:17:37.366 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:37.366 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 1.3 M 2024-01-21T03:17:37.366 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 5.5 M 2024-01-21T03:17:37.366 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-01-21T03:17:37.845 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-nbd-debuginfo-19.0.0-766.g0edf41a6.el9.x86_ 2.6 MB/s | 1.3 MB 00:00 2024-01-21T03:17:37.846 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:37.846 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.6 MB/s | 1.3 MB 00:00 2024-01-21T03:17:37.846 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-01-21T03:17:37.850 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-01-21T03:17:37.850 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-01-21T03:17:37.892 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-01-21T03:17:37.892 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-01-21T03:17:38.129 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-01-21T03:17:38.227 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:38.661 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:38.949 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-01-21T03:17:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:17:38.949 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-01-21T03:17:39.112 DEBUG:teuthology.parallel:result is None 2024-01-21T03:17:41.445 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:17:41.532 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:43.798 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:44.066 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:44.066 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:44.067 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:17:44.067 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:44.067 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:44.067 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:17:44.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs-debuginfo 2024-01-21T03:17:44.848 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:10:18 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:45.006 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:17:45.006 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:45.006 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:17:45.006 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:45.007 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:17:45.007 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 615 k 2024-01-21T03:17:45.007 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:45.007 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:17:45.007 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:45.007 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:17:45.007 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:45.008 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 615 k 2024-01-21T03:17:45.008 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.1 M 2024-01-21T03:17:45.008 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:17:45.285 INFO:teuthology.orchestra.run.smithi007.stdout:python3-cephfs-debuginfo-19.0.0-766.g0edf41a6.e 2.2 MB/s | 615 kB 00:00 2024-01-21T03:17:45.286 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:45.286 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.2 MB/s | 615 kB 00:00 2024-01-21T03:17:45.286 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:17:45.290 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:17:45.290 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:17:45.500 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:17:45.611 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T03:17:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T03:17:46.350 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T03:17:46.350 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:46.350 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:17:46.350 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:46.350 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:46.350 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:17:46.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados-debuginfo 2024-01-21T03:17:47.086 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:10:21 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:47.244 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:17:47.244 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:47.244 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:17:47.245 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:47.245 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:17:47.245 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 769 k 2024-01-21T03:17:47.245 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:47.245 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:17:47.245 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:47.245 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:17:47.245 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:47.245 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 769 k 2024-01-21T03:17:47.245 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.7 M 2024-01-21T03:17:47.245 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:17:48.380 INFO:teuthology.orchestra.run.smithi007.stdout:python3-rados-debuginfo-19.0.0-766.g0edf41a6.el 678 kB/s | 769 kB 00:01 2024-01-21T03:17:48.380 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:48.381 INFO:teuthology.orchestra.run.smithi007.stdout:Total 677 kB/s | 769 kB 00:01 2024-01-21T03:17:48.381 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:17:48.385 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:17:48.385 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:17:48.429 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:17:48.429 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:17:48.627 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:17:48.723 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x 1/1 2024-01-21T03:17:49.103 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x 1/1 2024-01-21T03:17:49.374 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x 1/1 2024-01-21T03:17:49.374 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:49.375 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:17:49.375 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:49.375 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:49.375 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:17:49.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd-debuginfo 2024-01-21T03:17:50.111 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:10:24 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:50.269 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:17:50.270 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:50.270 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:17:50.270 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:50.271 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:17:50.271 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.1 M 2024-01-21T03:17:50.271 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:50.271 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:17:50.271 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:50.271 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:17:50.271 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:50.271 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 1.1 M 2024-01-21T03:17:50.271 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 4.3 M 2024-01-21T03:17:50.271 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:17:50.600 INFO:teuthology.orchestra.run.smithi007.stdout:python3-rbd-debuginfo-19.0.0-766.g0edf41a6.el9. 3.4 MB/s | 1.1 MB 00:00 2024-01-21T03:17:50.600 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:50.601 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.4 MB/s | 1.1 MB 00:00 2024-01-21T03:17:50.601 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:17:50.604 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:17:50.604 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:17:50.645 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:17:50.645 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:17:50.872 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:17:50.992 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:17:51.453 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:17:51.739 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:17:51.739 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:51.740 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:17:51.740 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:51.740 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:51.740 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:17:51.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw-debuginfo 2024-01-21T03:17:52.510 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:10:26 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:52.668 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:17:52.669 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:52.669 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:17:52.669 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:52.669 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:17:52.669 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 337 k 2024-01-21T03:17:52.669 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:52.669 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:17:52.669 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:52.669 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:17:52.670 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:52.670 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 337 k 2024-01-21T03:17:52.670 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.0 M 2024-01-21T03:17:52.670 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:17:53.062 INFO:teuthology.orchestra.run.smithi007.stdout:python3-rgw-debuginfo-19.0.0-766.g0edf41a6.el9. 860 kB/s | 337 kB 00:00 2024-01-21T03:17:53.062 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:53.062 INFO:teuthology.orchestra.run.smithi007.stdout:Total 857 kB/s | 337 kB 00:00 2024-01-21T03:17:53.063 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:17:53.066 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:17:53.066 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:17:53.102 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:17:53.102 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:17:53.243 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:17:53.329 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:17:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:17:54.006 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T03:17:54.006 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:54.007 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:17:54.007 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:54.007 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:54.007 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:17:54.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse-debuginfo 2024-01-21T03:17:54.747 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:10:28 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:54.906 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:17:54.907 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:54.907 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:17:54.907 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:54.907 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:17:54.907 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 509 k 2024-01-21T03:17:54.907 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:54.907 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:17:54.907 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:54.907 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:17:54.908 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:54.908 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 509 k 2024-01-21T03:17:54.908 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.5 M 2024-01-21T03:17:54.908 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:17:55.349 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-debuginfo-19.0.0-766.g0edf41a6.el9.x86 1.1 MB/s | 509 kB 00:00 2024-01-21T03:17:55.350 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:55.350 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.1 MB/s | 509 kB 00:00 2024-01-21T03:17:55.350 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:17:55.353 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:17:55.354 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:17:55.391 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:17:55.392 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:17:55.559 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:17:55.676 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:56.078 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:56.356 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:17:56.356 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:56.356 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:17:56.356 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:17:56.356 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:56.356 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:17:56.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror-debuginfo 2024-01-21T03:17:57.112 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:10:31 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:17:57.269 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:17:57.270 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:57.270 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:17:57.270 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:57.271 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:17:57.271 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 52 M 2024-01-21T03:17:57.271 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:57.271 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:17:57.271 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:17:57.271 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:17:57.271 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:17:57.271 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 52 M 2024-01-21T03:17:57.271 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 298 M 2024-01-21T03:17:57.271 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:17:59.777 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-debuginfo-19.0.0-766.g0edf41a6.el9.x 21 MB/s | 52 MB 00:02 2024-01-21T03:17:59.778 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:17:59.778 INFO:teuthology.orchestra.run.smithi007.stdout:Total 21 MB/s | 52 MB 00:02 2024-01-21T03:17:59.778 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:17:59.782 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:17:59.783 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:18:00.133 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:18:00.133 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:18:06.412 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:18:06.502 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T03:18:08.625 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T03:18:08.930 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T03:18:08.930 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:08.930 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:18:08.930 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:18:08.930 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:08.930 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:18:09.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd-debuginfo 2024-01-21T03:18:09.702 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:10:43 ago on Sun 21 Jan 2024 03:07:26 AM UTC. 2024-01-21T03:18:09.860 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-01-21T03:18:09.861 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:18:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-01-21T03:18:09.861 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:18:09.861 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-01-21T03:18:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.3 M 2024-01-21T03:18:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:09.861 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-01-21T03:18:09.861 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-01-21T03:18:09.861 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-01-21T03:18:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:09.861 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 1.3 M 2024-01-21T03:18:09.861 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 5.5 M 2024-01-21T03:18:09.862 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-01-21T03:18:10.170 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-debuginfo-19.0.0-766.g0edf41a6.el9.x86_ 4.1 MB/s | 1.3 MB 00:00 2024-01-21T03:18:10.171 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-01-21T03:18:10.171 INFO:teuthology.orchestra.run.smithi007.stdout:Total 4.0 MB/s | 1.3 MB 00:00 2024-01-21T03:18:10.171 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-01-21T03:18:10.175 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-01-21T03:18:10.175 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-01-21T03:18:10.220 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-01-21T03:18:10.220 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-01-21T03:18:10.481 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-01-21T03:18:10.597 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:18:11.049 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:18:11.347 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T03:18:11.347 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:11.347 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-01-21T03:18:11.347 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T03:18:11.347 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:11.347 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-01-21T03:18:11.536 DEBUG:teuthology.parallel:result is None 2024-01-21T03:18:11.537 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T03:18:11.697 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T03:18:11.718 INFO:teuthology.orchestra.run.smithi007.stdout:19.0.0-766.g0edf41a6.el9 2024-01-21T03:18:11.718 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-766.g0edf41a6.el9 2024-01-21T03:18:11.718 INFO:teuthology.task.install:The correct ceph version 19.0.0-766.g0edf41a6 is installed. 2024-01-21T03:18:11.720 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T03:18:11.865 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T03:18:11.890 INFO:teuthology.orchestra.run.smithi177.stdout:19.0.0-766.g0edf41a6.el9 2024-01-21T03:18:11.890 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-766.g0edf41a6.el9 2024-01-21T03:18:11.890 INFO:teuthology.task.install:The correct ceph version 19.0.0-766.g0edf41a6 is installed. 2024-01-21T03:18:11.892 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-21T03:18:11.892 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:11.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T03:18:11.920 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:11.920 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T03:18:11.962 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-21T03:18:11.963 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:11.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T03:18:11.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T03:18:12.057 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:12.057 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T03:18:12.088 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T03:18:12.157 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-21T03:18:12.157 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:12.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T03:18:12.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T03:18:12.252 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:12.252 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T03:18:12.281 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T03:18:12.350 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-21T03:18:12.351 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:12.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T03:18:12.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T03:18:12.444 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:12.444 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T03:18:12.474 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T03:18:12.545 INFO:teuthology.run_tasks:Running task ceph... 2024-01-21T03:18:12.628 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-21T03:18:12.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /var/log/ceph 2024-01-21T03:18:12.630 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /var/log/ceph 2024-01-21T03:18:12.659 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-21T03:18:12.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-21T03:18:12.697 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-21T03:18:12.727 INFO:tasks.ceph:Creating extra log directories... 2024-01-21T03:18:12.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-21T03:18:12.764 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-21T03:18:12.805 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-21T03:18:12.805 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-21T03:18:12.805 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652', 'branch': 'wip-yuri8-testing-2024-01-18-0823', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7523652', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri8-testing-2024-01-18-0823'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri8-testing-2024-01-18-0823', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9599, 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri8-testing-2024-01-18-0823', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCysGnDiMG3hGIFgiXaTH/oCK9ndRzJ55SkMZMS2Q83b6pe6HQhs8gb6USSF4yFdUyIfEBXiIdb/fyyj27NdoiU=', 'smithi177.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCnBrRXFdjEgSX7q1KhDE3anvh06bdhYQXczN5WiULNIIELABIokQnA9prKtCgCRS7uekcklFQ8pNArUFnIQJHc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-19_16:13:55', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-21T03:18:12.806 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-21T03:18:12.831 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-21T03:18:12.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2024-01-21T03:18:12.887 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/run/ceph 2024-01-21T03:18:12.917 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:12.917 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-01-21T03:18:12.965 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-21T03:18:12.965 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-01-21T03:18:13.019 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-21T03:18:13.019 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T03:18:13.019 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-01-21T03:18:13.019 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T03:18:13.019 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T03:18:13.019 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-01-21 03:09:37.524846942 +0000 2024-01-21T03:18:13.020 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-01-21 03:06:39.562984802 +0000 2024-01-21T03:18:13.020 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-01-21 03:06:39.562984802 +0000 2024-01-21T03:18:13.020 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-01-21 03:06:39.562984802 +0000 2024-01-21T03:18:13.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-21T03:18:13.084 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-01-21T03:18:13.084 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-01-21T03:18:13.084 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.6555e-05 s, 5.3 MB/s 2024-01-21T03:18:13.085 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-21T03:18:13.140 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-01-21T03:18:13.194 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-21T03:18:13.194 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T03:18:13.194 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-01-21T03:18:13.194 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T03:18:13.195 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T03:18:13.195 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-01-21 03:09:37.525846928 +0000 2024-01-21T03:18:13.195 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-01-21 03:06:39.903979843 +0000 2024-01-21T03:18:13.195 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-01-21 03:06:39.903979843 +0000 2024-01-21T03:18:13.195 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-01-21 03:06:39.903979843 +0000 2024-01-21T03:18:13.195 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-21T03:18:13.259 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-01-21T03:18:13.259 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-01-21T03:18:13.259 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.3983e-05 s, 5.4 MB/s 2024-01-21T03:18:13.260 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-21T03:18:13.315 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-01-21T03:18:13.369 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-21T03:18:13.369 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T03:18:13.369 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-01-21T03:18:13.370 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T03:18:13.370 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T03:18:13.370 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-01-21 03:09:37.525846928 +0000 2024-01-21T03:18:13.370 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-01-21 03:06:40.215975304 +0000 2024-01-21T03:18:13.370 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-01-21 03:06:40.215975304 +0000 2024-01-21T03:18:13.370 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-01-21 03:06:40.215975304 +0000 2024-01-21T03:18:13.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-21T03:18:13.434 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-01-21T03:18:13.434 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-01-21T03:18:13.434 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 7.8414e-05 s, 6.5 MB/s 2024-01-21T03:18:13.436 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-21T03:18:13.491 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-01-21T03:18:13.545 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-21T03:18:13.545 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T03:18:13.545 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-01-21T03:18:13.545 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T03:18:13.546 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T03:18:13.546 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-01-21 03:09:37.526846913 +0000 2024-01-21T03:18:13.546 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-01-21 03:06:40.549970445 +0000 2024-01-21T03:18:13.546 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-01-21 03:06:40.549970445 +0000 2024-01-21T03:18:13.546 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-01-21 03:06:40.549970445 +0000 2024-01-21T03:18:13.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-21T03:18:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-01-21T03:18:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-01-21T03:18:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.5022e-05 s, 5.4 MB/s 2024-01-21T03:18:13.611 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-21T03:18:13.666 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-21T03:18:13.666 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:13.666 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-01-21T03:18:13.684 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-21T03:18:13.684 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-01-21T03:18:13.741 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-21T03:18:13.741 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T03:18:13.741 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-01-21T03:18:13.741 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T03:18:13.741 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T03:18:13.741 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-01-21 03:09:26.742036990 +0000 2024-01-21T03:18:13.741 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-01-21 03:06:34.212191794 +0000 2024-01-21T03:18:13.741 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-01-21 03:06:34.212191794 +0000 2024-01-21T03:18:13.741 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-01-21 03:06:34.212191794 +0000 2024-01-21T03:18:13.742 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-21T03:18:13.809 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-01-21T03:18:13.809 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-01-21T03:18:13.809 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000248483 s, 2.1 MB/s 2024-01-21T03:18:13.810 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-21T03:18:13.867 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-01-21T03:18:13.923 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-21T03:18:13.923 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T03:18:13.923 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-01-21T03:18:13.923 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T03:18:13.923 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T03:18:13.923 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-01-21 03:09:26.742036990 +0000 2024-01-21T03:18:13.923 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-01-21 03:06:34.553182172 +0000 2024-01-21T03:18:13.923 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-01-21 03:06:34.553182172 +0000 2024-01-21T03:18:13.923 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-01-21 03:06:34.553182172 +0000 2024-01-21T03:18:13.924 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-21T03:18:13.990 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-01-21T03:18:13.990 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-01-21T03:18:13.990 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000196513 s, 2.6 MB/s 2024-01-21T03:18:13.991 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-21T03:18:14.047 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-01-21T03:18:14.102 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-21T03:18:14.103 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T03:18:14.103 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-01-21T03:18:14.103 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T03:18:14.103 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T03:18:14.103 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-01-21 03:09:26.742036990 +0000 2024-01-21T03:18:14.103 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-01-21 03:06:34.906172211 +0000 2024-01-21T03:18:14.103 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-01-21 03:06:34.906172211 +0000 2024-01-21T03:18:14.103 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-01-21 03:06:34.906172211 +0000 2024-01-21T03:18:14.103 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-21T03:18:14.168 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-01-21T03:18:14.168 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-01-21T03:18:14.168 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000192239 s, 2.7 MB/s 2024-01-21T03:18:14.170 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-21T03:18:14.225 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-01-21T03:18:14.280 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-21T03:18:14.280 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T03:18:14.280 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-01-21T03:18:14.280 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T03:18:14.280 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T03:18:14.280 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-01-21 03:09:26.743036962 +0000 2024-01-21T03:18:14.281 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-01-21 03:06:35.288161431 +0000 2024-01-21T03:18:14.281 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-01-21 03:06:35.288161431 +0000 2024-01-21T03:18:14.281 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-01-21 03:06:35.288161431 +0000 2024-01-21T03:18:14.281 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-21T03:18:14.345 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-01-21T03:18:14.345 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-01-21T03:18:14.345 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000230734 s, 2.2 MB/s 2024-01-21T03:18:14.346 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-21T03:18:14.403 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-21T03:18:14.403 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-21T03:18:14.403 INFO:tasks.ceph:Generating config... 2024-01-21T03:18:14.404 INFO:tasks.ceph:[client] debug monc = 20 2024-01-21T03:18:14.404 INFO:tasks.ceph:[client] debug ms = 1 2024-01-21T03:18:14.404 INFO:tasks.ceph:[client] debug objecter = 20 2024-01-21T03:18:14.404 INFO:tasks.ceph:[client] debug rados = 20 2024-01-21T03:18:14.404 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-01-21T03:18:14.405 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-01-21T03:18:14.405 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-01-21T03:18:14.405 INFO:tasks.ceph:[global] ms type = async 2024-01-21T03:18:14.405 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-01-21T03:18:14.405 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-21T03:18:14.405 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-21T03:18:14.405 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-21T03:18:14.405 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-21T03:18:14.405 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-21T03:18:14.405 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-01-21T03:18:14.405 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-01-21T03:18:14.405 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-01-21T03:18:14.405 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-01-21T03:18:14.405 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-01-21T03:18:14.405 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-01-21T03:18:14.405 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-01-21T03:18:14.406 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] debug monc = 20 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-21T03:18:14.406 INFO:tasks.ceph:[osd] osd class default list = * 2024-01-21T03:18:14.407 INFO:tasks.ceph:[osd] osd class load list = * 2024-01-21T03:18:14.407 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-01-21T03:18:14.407 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-01-21T03:18:14.407 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-01-21T03:18:14.407 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-21T03:18:14.407 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-01-21T03:18:14.407 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-01-21T03:18:14.407 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-01-21T03:18:14.407 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-01-21T03:18:14.407 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-21T03:18:14.407 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-01-21T03:18:14.407 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-01-21T03:18:14.407 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-21T03:18:14.407 INFO:tasks.ceph:Setting up mon.a... 2024-01-21T03:18:14.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-21T03:18:14.460 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.keyring 2024-01-21T03:18:14.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-21T03:18:14.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-21T03:18:14.574 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.7'), ('mon.c', '[v2:172.21.15.7:3301,v1:172.21.15.7:6790]'), ('mon.b', '172.21.15.177')] 2024-01-21T03:18:14.575 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.7,[v2:172.21.15.7:3301,v1:172.21.15.7:6790],172.21.15.177', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-01-21T03:18:14.576 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:14.576 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-21T03:18:14.629 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.7 --addv c '[v2:172.21.15.7:3301,v1:172.21.15.7:6790]' --add b 172.21.15.177 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:14.712 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:14.713 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: generated fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:14.713 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = quincy 2024-01-21T03:18:14.713 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 0 2024-01-21T03:18:14.713 INFO:teuthology.orchestra.run.smithi007.stdout:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:14.713 INFO:teuthology.orchestra.run.smithi007.stdout:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:14.713 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:14.713 INFO:teuthology.orchestra.run.smithi007.stdout:min_mon_release 17 (quincy) 2024-01-21T03:18:14.713 INFO:teuthology.orchestra.run.smithi007.stdout:election_strategy: 1 2024-01-21T03:18:14.713 INFO:teuthology.orchestra.run.smithi007.stdout:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:18:14.713 INFO:teuthology.orchestra.run.smithi007.stdout:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:18:14.713 INFO:teuthology.orchestra.run.smithi007.stdout:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:18:14.713 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-21T03:18:14.714 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-21T03:18:14.768 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8de9e7dd-e982-4048-83dd-ee52350dd56d... 2024-01-21T03:18:14.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-21T03:18:14.811 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-21T03:18:14.858 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-01-21T03:18:14.858 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2024-01-21T03:18:14.858 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-21T03:18:14.858 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = cephx 2024-01-21T03:18:14.858 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.858 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2024-01-21T03:18:14.858 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.858 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2024-01-21T03:18:14.858 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.858 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2024-01-21T03:18:14.858 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2024-01-21T03:18:14.859 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2024-01-21T03:18:14.860 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = "172.21.15.7,[v2:172.21.15.7:3301,v1:172.21.15.7:6790],172.21.15.177" 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: mon client directed command retry = 5 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: mon election default strategy = 1 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject socket failures = 5000 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: ms type = async 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: osd heartbeat grace = 80 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2024-01-21T03:18:14.861 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug trim objects = true 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.862 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: bdev async discard = True 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: bdev enable discard = True 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore block size = 96636764160 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore fsck on mount = True 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore zero block detection = True 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 1 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 1 2024-01-21T03:18:14.863 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 20 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 20 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 10 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd backfillfull_ratio = 0.85 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd full ratio = 0.9 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd nearfull ratio = 0.8 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: osd class default list = * 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: osd class load list = * 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: osd client watch timeout = 120 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify cached snaps = True 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify missing on start = True 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: osd failsafe full ratio = 0.95 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: osd fast shutdown = False 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock override recovery settings = True 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: osd memory target = 1610612736 2024-01-21T03:18:14.864 INFO:teuthology.orchestra.run.smithi007.stdout: osd objectstore = bluestore 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout: osd op queue = debug_random 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout: osd op queue cut off = debug_random 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout: osd shutdown pgref assert = True 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2024-01-21T03:18:14.865 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 20 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2024-01-21T03:18:14.866 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: mon min osdmap epochs = 50 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd crush smoke test = False 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune interval = 2 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune min = 15 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune txsize = 2 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: mon scrub interval = 300 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on pool no app = False 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: paxos service trim min = 10 2024-01-21T03:18:14.867 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:18:14.868 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2024-01-21T03:18:14.868 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2024-01-21T03:18:14.868 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2024-01-21T03:18:14.868 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2024-01-21T03:18:14.868 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-21T03:18:14.868 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-21T03:18:14.868 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 20 2024-01-21T03:18:14.868 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-01-21T03:18:14.868 INFO:teuthology.orchestra.run.smithi007.stdout: debug objecter = 20 2024-01-21T03:18:14.868 INFO:teuthology.orchestra.run.smithi007.stdout: debug rados = 20 2024-01-21T03:18:14.868 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2024-01-21T03:18:14.868 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.c] 2024-01-21T03:18:14.868 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.b] 2024-01-21T03:18:14.869 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-01-21T03:18:14.869 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-01-21T03:18:14.869 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-21T03:18:14.869 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-01-21T03:18:14.869 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.869 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-01-21T03:18:14.869 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.869 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-01-21T03:18:14.869 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.869 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-01-21T03:18:14.869 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-01-21T03:18:14.869 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.869 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-01-21T03:18:14.870 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-01-21T03:18:14.871 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "172.21.15.7,[v2:172.21.15.7:3301,v1:172.21.15.7:6790],172.21.15.177" 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout: mon client directed command retry = 5 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout: mon election default strategy = 1 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject socket failures = 5000 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout: ms type = async 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout: osd heartbeat grace = 80 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-01-21T03:18:14.872 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.873 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: bdev async discard = True 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: bdev enable discard = True 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore zero block detection = True 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 1 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 1 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: debug monc = 20 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-01-21T03:18:14.874 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 10 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: osd class default list = * 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: osd class load list = * 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: osd client watch timeout = 120 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify cached snaps = True 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify missing on start = True 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: osd fast shutdown = False 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock override recovery settings = True 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: osd memory target = 1610612736 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-01-21T03:18:14.875 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue = debug_random 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue cut off = debug_random 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout: osd shutdown pgref assert = True 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-01-21T03:18:14.876 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-21T03:18:14.877 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: mon min osdmap epochs = 50 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd crush smoke test = False 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune interval = 2 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune min = 15 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune txsize = 2 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: mon scrub interval = 300 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on pool no app = False 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: paxos service trim min = 10 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-01-21T03:18:14.878 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-01-21T03:18:14.879 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-01-21T03:18:14.879 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-01-21T03:18:14.879 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-01-21T03:18:14.879 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-21T03:18:14.879 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-21T03:18:14.879 INFO:teuthology.orchestra.run.smithi177.stdout: debug monc = 20 2024-01-21T03:18:14.879 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-01-21T03:18:14.879 INFO:teuthology.orchestra.run.smithi177.stdout: debug objecter = 20 2024-01-21T03:18:14.879 INFO:teuthology.orchestra.run.smithi177.stdout: debug rados = 20 2024-01-21T03:18:14.879 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-01-21T03:18:14.879 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-01-21T03:18:14.879 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-01-21T03:18:14.884 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-21T03:18:14.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-21T03:18:14.979 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-21T03:18:14.979 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:14.979 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-21T03:18:14.993 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:14.993 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-21T03:18:15.047 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2024-01-21T03:18:15.048 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:15.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-21T03:18:15.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-21T03:18:15.127 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:15.127 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:15.181 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi177.front.sepia.ceph.com 2024-01-21T03:18:15.181 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:15.182 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-21T03:18:15.182 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-21T03:18:15.221 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:15.221 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:15.276 INFO:tasks.ceph:Setting up mon nodes... 2024-01-21T03:18:15.276 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-21T03:18:15.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-21T03:18:15.329 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-21T03:18:15.332 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-21T03:18:15.390 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-21T03:18:15.393 INFO:tasks.ceph:Setting up mds nodes... 2024-01-21T03:18:15.393 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-21T03:18:15.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-21T03:18:15.435 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-21T03:18:15.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-21T03:18:15.491 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-21T03:18:15.507 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-21T03:18:15.507 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-21T03:18:15.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-21T03:18:15.533 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-21T03:18:15.533 INFO:tasks.ceph:role: osd.0 2024-01-21T03:18:15.533 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com 2024-01-21T03:18:15.534 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-21T03:18:15.599 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T03:18:15.599 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T03:18:15.599 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T03:18:15.599 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T03:18:15.599 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T03:18:15.599 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-01-21T03:18:15.599 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T03:18:15.599 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T03:18:15.599 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T03:18:15.599 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T03:18:15.631 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-01-21T03:18:15.633 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-01-21T03:18:15.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-21T03:18:15.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-21T03:18:15.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-21T03:18:15.841 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-21T03:18:15.842 INFO:tasks.ceph:role: osd.1 2024-01-21T03:18:15.842 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com 2024-01-21T03:18:15.842 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-21T03:18:15.908 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T03:18:15.908 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T03:18:15.908 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T03:18:15.908 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T03:18:15.908 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T03:18:15.908 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-01-21T03:18:15.908 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T03:18:15.908 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T03:18:15.908 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T03:18:15.909 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T03:18:15.940 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-01-21T03:18:15.942 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-01-21T03:18:15.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-21T03:18:16.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-21T03:18:16.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-21T03:18:16.148 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-21T03:18:16.149 INFO:tasks.ceph:role: osd.2 2024-01-21T03:18:16.149 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com 2024-01-21T03:18:16.149 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-21T03:18:16.215 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T03:18:16.215 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T03:18:16.215 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T03:18:16.215 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T03:18:16.215 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T03:18:16.215 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-01-21T03:18:16.215 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T03:18:16.215 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T03:18:16.215 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T03:18:16.215 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T03:18:16.247 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-01-21T03:18:16.249 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-01-21T03:18:16.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-21T03:18:16.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-21T03:18:16.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-21T03:18:16.457 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-21T03:18:16.458 INFO:tasks.ceph:role: osd.3 2024-01-21T03:18:16.458 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com 2024-01-21T03:18:16.458 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-21T03:18:16.524 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T03:18:16.524 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T03:18:16.524 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T03:18:16.524 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T03:18:16.525 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T03:18:16.525 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-01-21T03:18:16.525 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T03:18:16.525 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T03:18:16.525 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T03:18:16.525 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T03:18:16.561 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-01-21T03:18:16.562 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-01-21T03:18:16.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-21T03:18:16.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-21T03:18:16.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:16.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:16.810+0000 7f5d25c0d4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-21T03:18:16.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:16.810+0000 7f5d25c0d4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-21T03:18:16.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:16.810+0000 7f5d25c0d4c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-21T03:18:16.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:16.810+0000 7f5d25c0d4c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-21T03:18:16.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:16.810+0000 7f5d25c0d4c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-21T03:18:16.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:16.810+0000 7f5d25c0d4c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-21T03:18:19.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-21T03:18:19.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:19.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:19.776+0000 7f5684c124c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-21T03:18:19.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:19.776+0000 7f5684c124c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-21T03:18:19.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:19.776+0000 7f5684c124c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-21T03:18:19.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:19.776+0000 7f5684c124c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-21T03:18:19.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:19.776+0000 7f5684c124c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-21T03:18:19.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:19.776+0000 7f5684c124c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-21T03:18:22.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-21T03:18:22.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:22.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:22.739+0000 7f5a310a34c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-21T03:18:22.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:22.739+0000 7f5a310a34c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-21T03:18:22.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:22.739+0000 7f5a310a34c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-21T03:18:22.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:22.739+0000 7f5a310a34c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-21T03:18:22.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:22.739+0000 7f5a310a34c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-21T03:18:22.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:22.740+0000 7f5a310a34c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-21T03:18:25.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-21T03:18:25.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:25.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:25.711+0000 7f8f7fbde4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-21T03:18:25.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:25.711+0000 7f8f7fbde4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-21T03:18:25.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:25.711+0000 7f8f7fbde4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-21T03:18:25.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:25.711+0000 7f8f7fbde4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-21T03:18:25.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:25.711+0000 7f8f7fbde4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-21T03:18:25.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:25.711+0000 7f8f7fbde4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-21T03:18:28.543 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-21T03:18:28.574 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-21T03:18:28.605 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-21T03:18:28.605 INFO:tasks.ceph:role: osd.4 2024-01-21T03:18:28.605 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com 2024-01-21T03:18:28.605 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-21T03:18:28.684 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T03:18:28.684 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T03:18:28.684 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T03:18:28.685 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T03:18:28.685 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T03:18:28.685 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-01-21T03:18:28.685 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T03:18:28.685 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T03:18:28.685 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T03:18:28.685 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T03:18:29.192 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-01-21T03:18:29.195 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-01-21T03:18:29.195 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-21T03:18:29.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-21T03:18:29.375 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-21T03:18:29.445 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-21T03:18:29.445 INFO:tasks.ceph:role: osd.5 2024-01-21T03:18:29.445 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com 2024-01-21T03:18:29.445 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-21T03:18:29.520 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T03:18:29.520 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T03:18:29.520 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T03:18:29.520 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T03:18:29.520 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T03:18:29.520 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-01-21T03:18:29.520 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T03:18:29.520 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T03:18:29.520 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T03:18:29.521 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T03:18:29.961 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-01-21T03:18:29.963 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-01-21T03:18:29.963 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-21T03:18:30.053 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-21T03:18:30.084 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-21T03:18:30.150 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-21T03:18:30.151 INFO:tasks.ceph:role: osd.6 2024-01-21T03:18:30.151 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com 2024-01-21T03:18:30.151 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-21T03:18:30.225 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T03:18:30.225 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T03:18:30.226 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T03:18:30.226 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T03:18:30.226 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T03:18:30.226 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-01-21T03:18:30.226 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T03:18:30.226 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T03:18:30.226 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T03:18:30.226 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T03:18:30.705 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-01-21T03:18:30.707 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-01-21T03:18:30.707 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-21T03:18:30.854 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-21T03:18:30.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-21T03:18:30.952 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-21T03:18:30.953 INFO:tasks.ceph:role: osd.7 2024-01-21T03:18:30.953 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com 2024-01-21T03:18:30.953 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-21T03:18:31.026 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T03:18:31.027 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T03:18:31.027 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T03:18:31.027 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T03:18:31.027 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T03:18:31.027 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-01-21T03:18:31.027 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T03:18:31.027 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T03:18:31.027 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T03:18:31.027 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T03:18:31.527 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-01-21T03:18:31.529 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-01-21T03:18:31.529 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-21T03:18:31.660 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-21T03:18:31.693 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:31.789 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:31.802+0000 7fea98fbc4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-21T03:18:31.789 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:31.802+0000 7fea98fbc4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-21T03:18:31.789 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:31.802+0000 7fea98fbc4c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-21T03:18:31.789 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:31.802+0000 7fea98fbc4c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-21T03:18:31.789 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:31.802+0000 7fea98fbc4c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-21T03:18:31.789 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:31.802+0000 7fea98fbc4c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-21T03:18:34.632 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-21T03:18:34.664 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:34.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:34.769+0000 7f8619c244c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-21T03:18:34.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:34.770+0000 7f8619c244c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-21T03:18:34.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:34.770+0000 7f8619c244c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-21T03:18:34.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:34.770+0000 7f8619c244c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-21T03:18:34.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:34.770+0000 7f8619c244c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-21T03:18:34.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:34.770+0000 7f8619c244c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-21T03:18:37.597 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-21T03:18:37.627 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:37.720 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:37.733+0000 7f14496144c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-21T03:18:37.720 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:37.734+0000 7f14496144c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-21T03:18:37.720 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:37.734+0000 7f14496144c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-21T03:18:37.720 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:37.734+0000 7f14496144c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-21T03:18:37.720 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:37.734+0000 7f14496144c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-21T03:18:37.721 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:37.734+0000 7f14496144c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-21T03:18:40.567 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-21T03:18:40.599 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:40.692 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:40.706+0000 7f604c4c94c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-21T03:18:40.693 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:40.706+0000 7f604c4c94c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-21T03:18:40.693 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:40.706+0000 7f604c4c94c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-21T03:18:40.693 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:40.706+0000 7f604c4c94c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-21T03:18:40.693 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:40.706+0000 7f604c4c94c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-21T03:18:40.693 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:40.707+0000 7f604c4c94c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-21T03:18:43.534 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-21T03:18:43.565 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-21T03:18:43.565 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:43.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-21T03:18:43.597 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:43.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-21T03:18:43.668 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:43.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-21T03:18:43.740 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:43.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-21T03:18:43.810 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:43.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-21T03:18:43.880 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:43.880 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-21T03:18:43.912 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:43.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-21T03:18:43.983 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:43.984 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-21T03:18:44.054 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:44.054 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-21T03:18:44.123 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:44.123 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-21T03:18:44.192 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:44.192 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-21T03:18:44.210 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:44.210 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-21T03:18:44.248 INFO:tasks.ceph:Adding keys to all mons... 2024-01-21T03:18:44.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-21T03:18:44.253 DEBUG:teuthology.orchestra.run.smithi177:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-21T03:18:44.291 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.y] 2024-01-21T03:18:44.291 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD3jKxl/jCJFBAAr3mLYoieLWywe+GNunrQUQ== 2024-01-21T03:18:44.291 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2024-01-21T03:18:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD4jKxluhhbMBAAutFMS2c1FYu3QFhY4mRdKw== 2024-01-21T03:18:44.292 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2024-01-21T03:18:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD7jKxlbGNSLhAA6fxuBkpVcR68R2WYpdm9/w== 2024-01-21T03:18:44.292 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2024-01-21T03:18:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD+jKxlZoMmLBAADk2JmHGDODA4xBzadePezw== 2024-01-21T03:18:44.292 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.3] 2024-01-21T03:18:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQABjaxlzQ90KhAApN/DvAi0Zs482U3QF+IbxA== 2024-01-21T03:18:44.292 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.x] 2024-01-21T03:18:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD3jKxlGWQrGBAAhy4vZZMcSCcGRNeV+XPtmw== 2024-01-21T03:18:44.292 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.4] 2024-01-21T03:18:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAHjaxlqNDdLxAAwuPx+B7KM3eSNy0szqvdKg== 2024-01-21T03:18:44.292 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.5] 2024-01-21T03:18:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAKjaxlIYryLRAAEFUc0BY/bcweDLMFVZ1kBA== 2024-01-21T03:18:44.293 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.6] 2024-01-21T03:18:44.293 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQANjaxl/VfKKxAAkGWj5Eckl8yTX1f5A6j/qw== 2024-01-21T03:18:44.293 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.7] 2024-01-21T03:18:44.293 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAQjaxlyVclKhAAdJXTP7gVPqD7q/4uJZC+OA== 2024-01-21T03:18:44.293 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-01-21T03:18:44.293 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD3jKxlq9nTGhAA851bPARNqfDAl/Pcx7iL+Q== 2024-01-21T03:18:44.293 INFO:teuthology.orchestra.run.smithi007.stdout:[client.1] 2024-01-21T03:18:44.293 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD3jKxl3WczHhAAORF5ntLL2cTQlOvPihg0/A== 2024-01-21T03:18:44.314 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.y] 2024-01-21T03:18:44.314 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQD3jKxl/jCJFBAAr3mLYoieLWywe+GNunrQUQ== 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.0] 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQD4jKxluhhbMBAAutFMS2c1FYu3QFhY4mRdKw== 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.1] 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQD7jKxlbGNSLhAA6fxuBkpVcR68R2WYpdm9/w== 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.2] 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQD+jKxlZoMmLBAADk2JmHGDODA4xBzadePezw== 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.3] 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQABjaxlzQ90KhAApN/DvAi0Zs482U3QF+IbxA== 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.x] 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQD3jKxlGWQrGBAAhy4vZZMcSCcGRNeV+XPtmw== 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.4] 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAHjaxlqNDdLxAAwuPx+B7KM3eSNy0szqvdKg== 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.5] 2024-01-21T03:18:44.315 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAKjaxlIYryLRAAEFUc0BY/bcweDLMFVZ1kBA== 2024-01-21T03:18:44.316 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.6] 2024-01-21T03:18:44.316 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQANjaxl/VfKKxAAkGWj5Eckl8yTX1f5A6j/qw== 2024-01-21T03:18:44.316 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.7] 2024-01-21T03:18:44.316 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAQjaxlyVclKhAAdJXTP7gVPqD7q/4uJZC+OA== 2024-01-21T03:18:44.316 INFO:teuthology.orchestra.run.smithi177.stdout:[client.0] 2024-01-21T03:18:44.316 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQD3jKxlq9nTGhAA851bPARNqfDAl/Pcx7iL+Q== 2024-01-21T03:18:44.316 INFO:teuthology.orchestra.run.smithi177.stdout:[client.1] 2024-01-21T03:18:44.316 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQD3jKxl3WczHhAAORF5ntLL2cTQlOvPihg0/A== 2024-01-21T03:18:44.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-21T03:18:44.337 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-21T03:18:44.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.468 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.522 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.577 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-21T03:18:44.631 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-21T03:18:44.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.685 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.738 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.792 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.846 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T03:18:44.897 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-21T03:18:44.899 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-21T03:18:44.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-21T03:18:44.956 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-21T03:18:45.010 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-21T03:18:45.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-21T03:18:45.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-21T03:18:45.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-21T03:18:45.159 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-21T03:18:45.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-21T03:18:45.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-21T03:18:45.349 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-21T03:18:45.377 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-21T03:18:45.476 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-21T03:18:45.503 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:45.505 DEBUG:teuthology.orchestra.run.smithi177:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-21T03:18:45.557 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-21T03:18:45.558 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-01-21T03:18:45.558 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-21T03:18:45.558 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-01-21T03:18:45.560 INFO:tasks.ceph.mon.a:Started 2024-01-21T03:18:45.560 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-01-21T03:18:45.561 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-21T03:18:45.561 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-01-21T03:18:45.563 INFO:tasks.ceph.mon.c:Started 2024-01-21T03:18:45.563 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-01-21T03:18:45.563 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-21T03:18:45.563 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-01-21T03:18:45.601 INFO:tasks.ceph.mon.b:Started 2024-01-21T03:18:45.601 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-21T03:18:45.601 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-21T03:18:45.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-01-21T03:18:45.605 INFO:tasks.ceph.mgr.y:Started 2024-01-21T03:18:45.605 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-21T03:18:45.605 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-01-21T03:18:45.607 INFO:tasks.ceph.mgr.x:Started 2024-01-21T03:18:45.607 INFO:tasks.ceph:Setting crush tunables to default 2024-01-21T03:18:45.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd crush tunables default 2024-01-21T03:18:45.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.724+0000 7f1151f41640 10 monclient: get_monmap_and_config 2024-01-21T03:18:45.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.724+0000 7f1151f41640 10 monclient: build_initial_monmap 2024-01-21T03:18:45.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.724+0000 7f1151f41640 10 monclient: monmap: 2024-01-21T03:18:45.711 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:18:45.711 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:45.711 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:45.725097+0000 2024-01-21T03:18:45.712 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:45.725097+0000 2024-01-21T03:18:45.712 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:18:45.712 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:18:45.712 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:18:45.712 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:18:45.712 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:18:45.712 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:18:45.712 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:18:45.712 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:18:45.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.724+0000 7f1151f41640 1 Processor -- start 2024-01-21T03:18:45.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.724+0000 7f1151f41640 1 -- start start 2024-01-21T03:18:45.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.724+0000 7f1151f41640 10 monclient: init 2024-01-21T03:18:45.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f1151f41640 10 monclient: _reopen_session rank -1 2024-01-21T03:18:45.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f1151f41640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-01-21T03:18:45.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f1151f41640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f114c0a8b20 0x7f114c145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:45.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f1151f41640 10 monclient(hunting): picked mon.noname-a con 0x7f114c0a8b20 addr v2:172.21.15.7:3300/0 2024-01-21T03:18:45.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f1151f41640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f114c0a9d10 addr v1:172.21.15.177:6789/0 2024-01-21T03:18:45.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f1151f41640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f114c0b2310 addr v1:172.21.15.7:6789/0 2024-01-21T03:18:45.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f1151f41640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c1456f0 con 0x7f114c0b2310 2024-01-21T03:18:45.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f1151f41640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c0a98c0 con 0x7f114c0a9d10 2024-01-21T03:18:45.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f1151f41640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:45.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f1151f41640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c1460c0 con 0x7f114c0a8b20 2024-01-21T03:18:45.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f1151f41640 10 monclient(hunting): _renew_subs 2024-01-21T03:18:45.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f1151f41640 10 monclient(hunting): authenticate will time out at 1272.041349s 2024-01-21T03:18:45.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f114bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f114c0b2310 legacy=0x7f114c14c850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-01-21T03:18:45.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f114b7fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f114c0a8b20 msgr2=0x7f114c145b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-01-21T03:18:45.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f114b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f114c0a8b20 0x7f114c145b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T03:18:45.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.725+0000 7f114affd640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f114c0a9d10 legacy=0x7f114c13f080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2024-01-21T03:18:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.926+0000 7f114b7fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f114c0a8b20 msgr2=0x7f114c145b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-01-21T03:18:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.926+0000 7f114bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f114c0b2310 legacy=0x7f114c14c850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-01-21T03:18:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.926+0000 7f114b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f114c0a8b20 0x7f114c145b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-21T03:18:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:45.926+0000 7f114affd640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f114c0a9d10 legacy=0x7f114c13f080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2024-01-21T03:18:46.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:46.327+0000 7f114b7fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f114c0a8b20 msgr2=0x7f114c145b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-01-21T03:18:46.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:46.327+0000 7f114bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f114c0b2310 legacy=0x7f114c14c850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-01-21T03:18:46.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:46.327+0000 7f114b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f114c0a8b20 0x7f114c145b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-21T03:18:46.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:46.327+0000 7f114affd640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f114c0a9d10 legacy=0x7f114c13f080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2024-01-21T03:18:47.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:47.128+0000 7f114b7fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f114c0a8b20 msgr2=0x7f114c145b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-01-21T03:18:47.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:47.128+0000 7f114bfff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f114c0b2310 legacy=0x7f114c14c850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-01-21T03:18:47.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:47.128+0000 7f114b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f114c0a8b20 0x7f114c145b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-21T03:18:47.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:47.128+0000 7f114affd640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f114c0a9d10 legacy=0x7f114c13f080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2024-01-21T03:18:48.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 10 monclient(hunting): tick 2024-01-21T03:18:48.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 10 monclient(hunting): _check_auth_tickets 2024-01-21T03:18:48.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 1 monclient(hunting): continuing hunt 2024-01-21T03:18:48.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 10 monclient(hunting): _reopen_session rank -1 2024-01-21T03:18:48.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f114c0a8b20 msgr2=0x7f114c145b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f114c0a8b20 0x7f114c145b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f114c0a9d10 legacy=0x7f114c13f080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f114c0b2310 legacy=0x7f114c14c850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 10 monclient: _add_conns ranks=[2,4] 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11380036a0 0x7f1138005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 10 monclient(hunting): picked mon.noname-b con 0x7f11380036a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1138006130 0x7f11380085f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 10 monclient(hunting): picked mon.noname-c con 0x7f1138006130 addr v2:172.21.15.177:3300/0 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c1460c0 con 0x7f1138006130 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1138008cb0 con 0x7f11380036a0 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f11497fa640 10 monclient(hunting): _renew_subs 2024-01-21T03:18:48.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f114b7fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11380036a0 msgr2=0x7f1138005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-01-21T03:18:48.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f114b7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11380036a0 0x7f1138005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T03:18:48.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f114affd640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7f1138006130 msgr2=0x7f11380085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.177:3300/0 2024-01-21T03:18:48.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.725+0000 7f114affd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1138006130 0x7f11380085f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T03:18:48.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.926+0000 7f114b7fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11380036a0 msgr2=0x7f1138005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-01-21T03:18:48.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.926+0000 7f114b7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11380036a0 0x7f1138005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-21T03:18:48.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.926+0000 7f114affd640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7f1138006130 msgr2=0x7f11380085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.177:3300/0 2024-01-21T03:18:48.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:48.926+0000 7f114affd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1138006130 0x7f11380085f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-21T03:18:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:49.327+0000 7f114b7fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11380036a0 msgr2=0x7f1138005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-01-21T03:18:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:49.327+0000 7f114b7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11380036a0 0x7f1138005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-21T03:18:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:49.327+0000 7f114affd640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7f1138006130 msgr2=0x7f11380085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.177:3300/0 2024-01-21T03:18:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:49.327+0000 7f114affd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1138006130 0x7f11380085f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-21T03:18:50.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:50.128+0000 7f114b7fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11380036a0 msgr2=0x7f1138005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-01-21T03:18:50.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:50.128+0000 7f114b7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11380036a0 0x7f1138005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-21T03:18:50.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:50.128+0000 7f114affd640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7f1138006130 msgr2=0x7f11380085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.177:3300/0 2024-01-21T03:18:50.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:50.128+0000 7f114affd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1138006130 0x7f11380085f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-21T03:18:51.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 10 monclient(hunting): tick 2024-01-21T03:18:51.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 10 monclient(hunting): _check_auth_tickets 2024-01-21T03:18:51.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 1 monclient(hunting): continuing hunt 2024-01-21T03:18:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 10 monclient(hunting): _reopen_session rank -1 2024-01-21T03:18:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11380036a0 msgr2=0x7f1138005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T03:18:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11380036a0 0x7f1138005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7f1138006130 msgr2=0x7f11380085f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T03:18:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1138006130 0x7f11380085f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-01-21T03:18:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f114bfff640 1 -- reap_dead start 2024-01-21T03:18:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f114c0a9d10 addr v1:172.21.15.177:6789/0 2024-01-21T03:18:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1138003ae0 addr v1:172.21.15.7:6789/0 2024-01-21T03:18:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1138007d00 0x7f1138008180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 10 monclient(hunting): picked mon.noname-a con 0x7f1138007d00 addr v2:172.21.15.7:3300/0 2024-01-21T03:18:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c146200 con 0x7f1138003ae0 2024-01-21T03:18:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c1456f0 con 0x7f114c0a9d10 2024-01-21T03:18:51.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:51.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c1460c0 con 0x7f1138007d00 2024-01-21T03:18:51.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f11497fa640 10 monclient(hunting): _renew_subs 2024-01-21T03:18:51.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f114affd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f1138003ae0 legacy=0x7f11380065e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-01-21T03:18:51.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f114bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f1138007d00 msgr2=0x7f1138008180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-01-21T03:18:51.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.725+0000 7f114bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1138007d00 0x7f1138008180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T03:18:51.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.726+0000 7f114b7fe640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f114c0a9d10 legacy=0x7f113800af60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2024-01-21T03:18:51.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.926+0000 7f114bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f1138007d00 msgr2=0x7f1138008180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-01-21T03:18:51.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.926+0000 7f114affd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f1138003ae0 legacy=0x7f11380065e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-01-21T03:18:51.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.926+0000 7f114bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1138007d00 0x7f1138008180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-21T03:18:51.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:51.927+0000 7f114b7fe640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f114c0a9d10 legacy=0x7f113800af60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2024-01-21T03:18:52.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:52.327+0000 7f114affd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f1138003ae0 legacy=0x7f11380065e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-01-21T03:18:52.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:52.327+0000 7f114bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f1138007d00 msgr2=0x7f1138008180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-01-21T03:18:52.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:52.327+0000 7f114bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1138007d00 0x7f1138008180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-21T03:18:52.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:52.328+0000 7f114b7fe640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f114c0a9d10 legacy=0x7f113800af60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2024-01-21T03:18:52.526 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:00:06.799 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:18:52.833 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:00:07.107 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:18:52.979 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:00:07.308 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:18:52.980 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:00:07.308 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:18:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:53.128+0000 7f114affd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f1138003ae0 legacy=0x7f11380065e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-01-21T03:18:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:53.128+0000 7f114bfff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f1138007d00 msgr2=0x7f1138008180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-01-21T03:18:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:53.128+0000 7f114bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1138007d00 0x7f1138008180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-21T03:18:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:53.129+0000 7f114b7fe640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f114c0a9d10 legacy=0x7f113800af60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2024-01-21T03:18:53.307 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:00:07.636 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:18:53.307 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:00:07.636 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:18:54.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 10 monclient(hunting): tick 2024-01-21T03:18:54.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 10 monclient(hunting): _check_auth_tickets 2024-01-21T03:18:54.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 1 monclient(hunting): continuing hunt 2024-01-21T03:18:54.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 10 monclient(hunting): _reopen_session rank -1 2024-01-21T03:18:54.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7f1138007d00 msgr2=0x7f1138008180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T03:18:54.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1138007d00 0x7f1138008180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f114c0a9d10 legacy=0x7f113800af60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T03:18:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f1138003ae0 legacy=0x7f11380065e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T03:18:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 10 monclient: _add_conns ranks=[4,2] 2024-01-21T03:18:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 10 monclient(hunting): picked mon.noname-c con 0x7f113800e690 addr v2:172.21.15.177:3300/0 2024-01-21T03:18:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 10 monclient(hunting): picked mon.noname-b con 0x7f1138010fc0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:18:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c1460c0 con 0x7f113800e690 2024-01-21T03:18:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1138008cb0 con 0x7f1138010fc0 2024-01-21T03:18:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.726+0000 7f11497fa640 10 monclient(hunting): _renew_subs 2024-01-21T03:18:54.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.734+0000 7f114b7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:54.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.749+0000 7f114b7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:59934/0 (socket says 172.21.15.7:59934) 2024-01-21T03:18:54.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.749+0000 7f114b7fe640 1 -- 172.21.15.7:0/3583569641 learned_addr learned my addr 172.21.15.7:0/3583569641 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:18:54.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.749+0000 7f114b7fe640 10 monclient(hunting): get_auth_request con 0x7f113800e690 auth_method 0 2024-01-21T03:18:54.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.749+0000 7f114b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:54.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.749+0000 7f114b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:54.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.749+0000 7f114b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:18:54.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.778+0000 7f114b7fe640 1 -- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 msgr2=0x7f1138010a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-21T03:18:54.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.778+0000 7f114b7fe640 1 -- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 msgr2=0x7f1138010a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T03:18:54.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.778+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T03:18:54.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.778+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T03:18:54.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.824+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:54.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.980+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:54.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.980+0000 7f114b7fe640 10 monclient(hunting): get_auth_request con 0x7f113800e690 auth_method 0 2024-01-21T03:18:54.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.980+0000 7f114b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:54.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.980+0000 7f114b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:54.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.980+0000 7f114b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:18:54.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.981+0000 7f114b7fe640 1 -- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 msgr2=0x7f1138010a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-21T03:18:54.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.981+0000 7f114b7fe640 1 -- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 msgr2=0x7f1138010a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T03:18:54.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.981+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T03:18:54.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:54.981+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-21T03:18:54.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.000+0000 7f114affd640 10 monclient(hunting): get_auth_request con 0x7f1138010fc0 auth_method 0 2024-01-21T03:18:54.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.000+0000 7f114affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:54.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.000+0000 7f114affd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:54.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.000+0000 7f114affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:18:55.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.056+0000 7f114affd640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 msgr2=0x7f11380133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-21T03:18:55.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.056+0000 7f114affd640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 msgr2=0x7f11380133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T03:18:55.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.056+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T03:18:55.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.057+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T03:18:55.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.433+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:55.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.433+0000 7f114b7fe640 10 monclient(hunting): get_auth_request con 0x7f113800e690 auth_method 0 2024-01-21T03:18:55.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.433+0000 7f114b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:55.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.433+0000 7f114b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:55.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.433+0000 7f114b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:18:55.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.437+0000 7f114b7fe640 1 -- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 msgr2=0x7f1138010a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-21T03:18:55.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.437+0000 7f114b7fe640 1 -- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 msgr2=0x7f1138010a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T03:18:55.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.437+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T03:18:55.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.437+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-21T03:18:55.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.509+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:55.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.510+0000 7f114affd640 10 monclient(hunting): get_auth_request con 0x7f1138010fc0 auth_method 0 2024-01-21T03:18:55.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.510+0000 7f114affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:55.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.510+0000 7f114affd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:55.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.510+0000 7f114affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:18:55.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.512+0000 7f114affd640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 msgr2=0x7f11380133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-21T03:18:55.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.512+0000 7f114affd640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 msgr2=0x7f11380133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T03:18:55.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.512+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T03:18:55.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.512+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-21T03:18:55.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.953+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:55.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.986+0000 7f114affd640 10 monclient(hunting): get_auth_request con 0x7f1138010fc0 auth_method 0 2024-01-21T03:18:55.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.986+0000 7f114affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:55.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.986+0000 7f114affd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:55.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:55.986+0000 7f114affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:18:55.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.006+0000 7f114affd640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 msgr2=0x7f11380133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-21T03:18:55.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.006+0000 7f114affd640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 msgr2=0x7f11380133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T03:18:55.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.006+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T03:18:55.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.006+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-21T03:18:56.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.238+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:56.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.239+0000 7f114b7fe640 10 monclient(hunting): get_auth_request con 0x7f113800e690 auth_method 0 2024-01-21T03:18:56.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.239+0000 7f114b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:56.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.239+0000 7f114b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:56.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.239+0000 7f114b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:18:56.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.239+0000 7f114b7fe640 1 -- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 msgr2=0x7f1138010a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-21T03:18:56.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.239+0000 7f114b7fe640 1 -- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 msgr2=0x7f1138010a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T03:18:56.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.239+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T03:18:56.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.239+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-21T03:18:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.807+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.808+0000 7f114affd640 10 monclient(hunting): get_auth_request con 0x7f1138010fc0 auth_method 0 2024-01-21T03:18:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.808+0000 7f114affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.808+0000 7f114affd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.808+0000 7f114affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:18:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.809+0000 7f114affd640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 msgr2=0x7f11380133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-21T03:18:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.809+0000 7f114affd640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 msgr2=0x7f11380133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T03:18:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.809+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T03:18:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:56.809+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-21T03:18:57.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 10 monclient(hunting): tick 2024-01-21T03:18:57.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 10 monclient(hunting): _check_auth_tickets 2024-01-21T03:18:57.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 1 monclient(hunting): continuing hunt 2024-01-21T03:18:57.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 10 monclient(hunting): _reopen_session rank -1 2024-01-21T03:18:57.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 msgr2=0x7f11380133b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T03:18:57.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138010fc0 0x7f11380133b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:57.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 1 -- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 msgr2=0x7f1138010a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T03:18:57.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 1 --2- 172.21.15.7:0/3583569641 >> v2:172.21.15.177:3300/0 conn(0x7f113800e690 0x7f1138010a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:57.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f114bfff640 1 -- 172.21.15.7:0/3583569641 reap_dead start 2024-01-21T03:18:57.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-01-21T03:18:57.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 0x7f1138008120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:57.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 10 monclient(hunting): picked mon.noname-b con 0x7f1138003ae0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:18:57.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1138006510 addr v1:172.21.15.177:6789/0 2024-01-21T03:18:57.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1138010a80 addr v1:172.21.15.7:6789/0 2024-01-21T03:18:57.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 1 -- 172.21.15.7:0/3583569641 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c0a98c0 con 0x7f1138010a80 2024-01-21T03:18:57.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 1 -- 172.21.15.7:0/3583569641 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c146200 con 0x7f1138006510 2024-01-21T03:18:57.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:57.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 1 -- 172.21.15.7:0/3583569641 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114c1460c0 con 0x7f1138003ae0 2024-01-21T03:18:57.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f11497fa640 10 monclient(hunting): _renew_subs 2024-01-21T03:18:57.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.726+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 0x7f1138008120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:57.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.728+0000 7f114b7fe640 10 monclient(hunting): get_auth_request con 0x7f1138003ae0 auth_method 0 2024-01-21T03:18:57.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.728+0000 7f114b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:57.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.728+0000 7f114b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:57.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.728+0000 7f114b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:18:57.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.729+0000 7f114b7fe640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 msgr2=0x7f1138008120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-21T03:18:57.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.729+0000 7f114b7fe640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 msgr2=0x7f1138008120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T03:18:57.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.729+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 0x7f1138008120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T03:18:57.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:57.729+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 0x7f1138008120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T03:18:58.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.126+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 0x7f1138008120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:58.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.133+0000 7f114b7fe640 10 monclient(hunting): get_auth_request con 0x7f1138003ae0 auth_method 0 2024-01-21T03:18:58.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.133+0000 7f114b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:58.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.133+0000 7f114b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:58.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.133+0000 7f114b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:18:58.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.133+0000 7f114b7fe640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 msgr2=0x7f1138008120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-21T03:18:58.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.133+0000 7f114b7fe640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 msgr2=0x7f1138008120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T03:18:58.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.133+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 0x7f1138008120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T03:18:58.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.134+0000 7f114b7fe640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 0x7f1138008120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-21T03:18:58.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.427+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427984783 0 0) 0x7f114c0a98c0 con 0x7f1138010a80 2024-01-21T03:18:58.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.427+0000 7f114a7fc640 10 monclient(hunting): _negotiate 2024-01-21T03:18:58.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.427+0000 7f114a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:58.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.427+0000 7f114a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:18:58.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.427+0000 7f114a7fc640 10 monclient(hunting): my global_id is 4098 2024-01-21T03:18:58.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.427+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1134003520 con 0x7f1138010a80 2024-01-21T03:18:58.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.459+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445707108 0 0) 0x7f114c146200 con 0x7f1138006510 2024-01-21T03:18:58.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.459+0000 7f114a7fc640 10 monclient(hunting): _negotiate 2024-01-21T03:18:58.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.459+0000 7f114a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:58.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.459+0000 7f114a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:18:58.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.459+0000 7f114a7fc640 10 monclient(hunting): my global_id is 4102 2024-01-21T03:18:58.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.459+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f114c0a98c0 con 0x7f1138006510 2024-01-21T03:18:58.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.486+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1079497985 0 0) 0x7f1134003520 con 0x7f1138010a80 2024-01-21T03:18:58.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.486+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 msgr2=0x7f1138008120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T03:18:58.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.486+0000 7f114a7fc640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 0x7f1138008120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:58.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 >> v1:172.21.15.177:6789/0 conn(0x7f1138006510 legacy=0x7f11380068f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient: _finish_hunting 0 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 1 monclient: found mon.noname-a-legacy 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.7:6789/0 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c1456f0 con 0x7f1138010a80 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient: _finish_auth 0 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient: _check_auth_tickets 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 749591385 0 0) 0x7f1144004140 con 0x7f1138010a80 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f1151f41640 5 monclient: authenticate success, global_id 4098 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient: dump: 2024-01-21T03:18:58.475 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] but i'm connected to v1:172.21.15.7:6789/0, reconnecting 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient: _reopen_session rank -1 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 >> v1:172.21.15.7:6789/0 conn(0x7f1138010a80 legacy=0x7f1138008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:18:58.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1134003b30 0x7f1134003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient(hunting): picked mon.a con 0x7f1134003b30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11340047c0 0x7f1134004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient(hunting): picked mon.b con 0x7f11340047c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 0x7f1134005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient(hunting): picked mon.c con 0x7f1138003ae0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c1460c0 con 0x7f1134003b30 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113400bb40 con 0x7f11340047c0 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113400bd00 con 0x7f1138003ae0 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient(hunting): _renew_subs 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114a7fc640 10 monclient(hunting): _finish_auth 0 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f1151f41640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-01-21T03:18:58.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.487+0000 7f114bfff640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 0x7f1134005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:58.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.488+0000 7f114bfff640 10 monclient(hunting): get_auth_request con 0x7f1138003ae0 auth_method 0 2024-01-21T03:18:58.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.488+0000 7f114bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:58.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.488+0000 7f114bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:58.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.488+0000 7f114bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:18:58.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.496+0000 7f114bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:18:58.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.496+0000 7f114bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:18:58.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.496+0000 7f114bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T03:18:58.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.516+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11340047c0 0x7f1134004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.522+0000 7f114affd640 10 monclient(hunting): get_auth_request con 0x7f11340047c0 auth_method 0 2024-01-21T03:18:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.522+0000 7f114affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.522+0000 7f114affd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.522+0000 7f114affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:18:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.527+0000 7f114affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:18:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.527+0000 7f114affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:18:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.527+0000 7f114affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T03:18:58.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.540+0000 7f114affd640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-01-21T03:18:58.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.541+0000 7f114affd640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 msgr2=0x7f1134005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:18:58.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.541+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1138003ae0 0x7f1134005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:58.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.541+0000 7f114affd640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1134003b30 msgr2=0x7f1134003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:18:58.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.541+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1134003b30 0x7f1134003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:58.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.541+0000 7f114affd640 10 monclient: _finish_hunting 0 2024-01-21T03:18:58.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.541+0000 7f114affd640 1 monclient: found mon.b 2024-01-21T03:18:58.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.541+0000 7f114affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:18:58.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.541+0000 7f114affd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:18:58.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.541+0000 7f114affd640 1 -- 172.21.15.7:0/3583569641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1134003520 con 0x7f11340047c0 2024-01-21T03:18:58.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.541+0000 7f114affd640 10 monclient: _finish_auth 0 2024-01-21T03:18:58.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.541+0000 7f114affd640 10 monclient: _check_auth_tickets 2024-01-21T03:18:58.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.541+0000 7f114affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:18:58.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.572+0000 7f114affd640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11340047c0 0x7f1134004c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1140003c20 tx=0x7f11340057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a0535100fa9d926 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:18:58.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.616+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1140031650 con 0x7f11340047c0 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.616+0000 7f114a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.616+0000 7f114a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.616+0000 7f114a7fc640 10 monclient: dump: 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:18:58.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.616+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11400317d0 con 0x7f11340047c0 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.616+0000 7f1151f41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.616+0000 7f114a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.616+0000 7f114a7fc640 1 -- 172.21.15.7:0/3583569641 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1140031b80 con 0x7f11340047c0 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.616+0000 7f1151f41640 10 monclient: get_monmap_and_config success 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.616+0000 7f1151f41640 4 set_mon_vals no callback set 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.616+0000 7f114a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.616+0000 7f114a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.616+0000 7f114a7fc640 10 monclient: dump: 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:18:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:18:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:18:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:18:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:18:58.605 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:18:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.617+0000 7f1151f41640 10 monclient: shutdown 2024-01-21T03:18:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.617+0000 7f1151f41640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:18:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.617+0000 7f1151f41640 1 -- 172.21.15.7:0/3583569641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11340047c0 msgr2=0x7f1134004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:18:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.617+0000 7f1151f41640 1 --2- 172.21.15.7:0/3583569641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11340047c0 0x7f1134004c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1140003c20 tx=0x7f11340057f0 comp rx=0 tx=0).stop 2024-01-21T03:18:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.617+0000 7f114bfff640 1 -- 172.21.15.7:0/3583569641 reap_dead start 2024-01-21T03:18:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.617+0000 7f1151f41640 1 -- 172.21.15.7:0/3583569641 shutdown_connections 2024-01-21T03:18:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.617+0000 7f1151f41640 1 -- 172.21.15.7:0/3583569641 >> 172.21.15.7:0/3583569641 conn(0x7f114c066fb0 msgr2=0x7f114c0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:18:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.617+0000 7f1151f41640 1 -- 172.21.15.7:0/3583569641 shutdown_connections 2024-01-21T03:18:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.617+0000 7f1151f41640 1 -- 172.21.15.7:0/3583569641 wait complete. 2024-01-21T03:18:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.618+0000 7f1151f41640 10 monclient: build_initial_monmap 2024-01-21T03:18:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.618+0000 7f1151f41640 10 monclient: monmap: 2024-01-21T03:18:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:18:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:18:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:18:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:18:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:18:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:18:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:18:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:18:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:18:58.609 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:18:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.619+0000 7f1151f41640 1 librados: starting msgr at 2024-01-21T03:18:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.619+0000 7f1151f41640 1 librados: starting objecter 2024-01-21T03:18:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.619+0000 7f1151f41640 1 Processor -- start 2024-01-21T03:18:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.619+0000 7f1151f41640 1 -- start start 2024-01-21T03:18:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.619+0000 7f1151f41640 1 librados: setting wanted keys 2024-01-21T03:18:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.619+0000 7f1151f41640 1 librados: calling monclient init 2024-01-21T03:18:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.619+0000 7f1151f41640 10 monclient: init 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 10 monclient: _reopen_session rank -1 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f114c1433f0 0x7f114c1d7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 10 monclient(hunting): picked mon.noname-c con 0x7f114c1433f0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f114c142d40 0x7f114c1da7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 10 monclient(hunting): picked mon.noname-a con 0x7f114c142d40 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f114c13fd60 0x7f114c1dad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 10 monclient(hunting): picked mon.noname-b con 0x7f114c13fd60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c0ac020 con 0x7f114c142d40 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c1437d0 con 0x7f114c13fd60 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114c1d74f0 con 0x7f114c1433f0 2024-01-21T03:18:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 10 monclient(hunting): _renew_subs 2024-01-21T03:18:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.620+0000 7f1151f41640 10 monclient(hunting): authenticate will time out at 1284.936267s 2024-01-21T03:18:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.621+0000 7f114bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f114c13fd60 0x7f114c1dad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.621+0000 7f114bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f114c13fd60 0x7f114c1dad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:38586/0 (socket says 172.21.15.7:38586) 2024-01-21T03:18:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.621+0000 7f114bfff640 1 -- 172.21.15.7:0/4046902896 learned_addr learned my addr 172.21.15.7:0/4046902896 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:18:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.621+0000 7f114bfff640 10 monclient(hunting): get_auth_request con 0x7f114c13fd60 auth_method 0 2024-01-21T03:18:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.621+0000 7f114bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.621+0000 7f114bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.621+0000 7f114bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:18:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.621+0000 7f114bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:18:58.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.621+0000 7f114bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:18:58.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.622+0000 7f114bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:18:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.632+0000 7f114bfff640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2024-01-21T03:18:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.632+0000 7f114bfff640 1 -- 172.21.15.7:0/4046902896 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f114c1433f0 msgr2=0x7f114c1d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:18:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.632+0000 7f114bfff640 1 --2- 172.21.15.7:0/4046902896 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f114c1433f0 0x7f114c1d7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.632+0000 7f114bfff640 1 -- 172.21.15.7:0/4046902896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f114c142d40 msgr2=0x7f114c1da7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:18:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.632+0000 7f114bfff640 1 --2- 172.21.15.7:0/4046902896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f114c142d40 0x7f114c1da7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.632+0000 7f114bfff640 10 monclient: _finish_hunting 0 2024-01-21T03:18:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.632+0000 7f114bfff640 1 monclient: found mon.noname-b 2024-01-21T03:18:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.632+0000 7f114bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:18:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.632+0000 7f114bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:18:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.632+0000 7f114bfff640 1 -- 172.21.15.7:0/4046902896 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c1439a0 con 0x7f114c13fd60 2024-01-21T03:18:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.636+0000 7f114bfff640 1 --2- 172.21.15.7:0/4046902896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f114c13fd60 0x7f114c1dad00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1144003d00 tx=0x7f114400fc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:18:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.636+0000 7f1133fff640 10 client.?.objecter ms_handle_connect 0x7f114c13fd60 2024-01-21T03:18:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.636+0000 7f1133fff640 10 client.?.objecter resend_mon_ops 2024-01-21T03:18:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1133fff640 1 -- 172.21.15.7:0/4046902896 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114400fec0 con 0x7f114c13fd60 2024-01-21T03:18:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1133fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:18:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1133fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:18:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1133fff640 10 monclient: dump: 2024-01-21T03:18:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:18:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:18:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:18:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1133fff640 10 monclient: _finish_auth 0 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1133fff640 10 monclient: _check_auth_tickets 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1133fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1133fff640 1 -- 172.21.15.7:0/4046902896 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114400d860 con 0x7f114c13fd60 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1133fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1133fff640 1 -- 172.21.15.7:0/4046902896 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114400dc10 con 0x7f114c13fd60 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1151f41640 5 monclient: authenticate success, global_id 4108 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1151f41640 10 monclient: _renew_subs 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1151f41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1151f41640 1 -- 172.21.15.7:0/4046902896 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114c1424d0 con 0x7f114c13fd60 2024-01-21T03:18:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1133fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1133fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1133fff640 10 monclient: dump: 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1149ffb640 4 set_mon_vals no callback set 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1151f41640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:18:58.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1151f41640 10 monclient: _renew_subs 2024-01-21T03:18:58.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1151f41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:18:58.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1151f41640 1 -- 172.21.15.7:0/4046902896 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f114c1d80a0 con 0x7f114c13fd60 2024-01-21T03:18:58.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.640+0000 7f1151f41640 1 librados: init done 2024-01-21T03:18:58.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.641+0000 7f1151f41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:18:58.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.641+0000 7f1151f41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:18:58.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.641+0000 7f1151f41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:18:58.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.641+0000 7f1151f41640 1 -- 172.21.15.7:0/4046902896 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f114c1d72c0 con 0x7f114c13fd60 2024-01-21T03:18:58.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.661+0000 7f1133fff640 1 -- 172.21.15.7:0/4046902896 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f114400dd70 con 0x7f114c13fd60 2024-01-21T03:18:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.661+0000 7f1133fff640 1 -- 172.21.15.7:0/4046902896 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1144028550 con 0x7f114c13fd60 2024-01-21T03:18:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.661+0000 7f1133fff640 10 client.4108.objecter ms_dispatch 0x7f114c000b90 osd_map(1..1 src has 1..1) v4 2024-01-21T03:18:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.661+0000 7f1133fff640 3 client.4108.objecter handle_osd_map got epochs [1,1] > 0 2024-01-21T03:18:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.661+0000 7f1133fff640 3 client.4108.objecter handle_osd_map decoding full epoch 1 2024-01-21T03:18:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.661+0000 7f1133fff640 20 client.4108.objecter dump_active .. 0 homeless 2024-01-21T03:18:58.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.829+0000 7f1133fff640 1 -- 172.21.15.7:0/4046902896 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f1144028810 con 0x7f114c13fd60 2024-01-21T03:18:58.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.829+0000 7f1133fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:18:58.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.829+0000 7f1133fff640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:18:58.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.897+0000 7f1151f41640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-01-21T03:18:58.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.897+0000 7f1151f41640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-01-21T03:18:58.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.897+0000 7f1151f41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:18:58.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:58.897+0000 7f1151f41640 1 -- 172.21.15.7:0/4046902896 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f114c04e970 con 0x7f114c13fd60 2024-01-21T03:18:59.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.189+0000 7f1133fff640 1 -- 172.21.15.7:0/4046902896 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f11440289b0 con 0x7f114c13fd60 2024-01-21T03:18:59.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.189+0000 7f1133fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-01-21T03:18:59.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.189+0000 7f1133fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-01-21T03:18:59.176 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-01-21T03:18:59.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 10 librados: watch_flush enter 2024-01-21T03:18:59.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 10 librados: watch_flush exit 2024-01-21T03:18:59.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 20 client.4108.objecter shutdown clearing up homeless session... 2024-01-21T03:18:59.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 10 client.4108.objecter successfully canceled tick 2024-01-21T03:18:59.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 10 monclient: shutdown 2024-01-21T03:18:59.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:18:59.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 1 -- 172.21.15.7:0/4046902896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f114c13fd60 msgr2=0x7f114c1dad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:18:59.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 1 --2- 172.21.15.7:0/4046902896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f114c13fd60 0x7f114c1dad00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1144003d00 tx=0x7f114400fc10 comp rx=0 tx=0).stop 2024-01-21T03:18:59.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 1 -- 172.21.15.7:0/4046902896 shutdown_connections 2024-01-21T03:18:59.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 1 --2- 172.21.15.7:0/4046902896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f114c13fd60 0x7f114c1dad00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:59.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 1 --2- 172.21.15.7:0/4046902896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f114c142d40 0x7f114c1da7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:59.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 1 --2- 172.21.15.7:0/4046902896 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f114c1433f0 0x7f114c1d7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:59.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 1 -- 172.21.15.7:0/4046902896 >> 172.21.15.7:0/4046902896 conn(0x7f114c066fb0 msgr2=0x7f114c13e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:18:59.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 1 -- 172.21.15.7:0/4046902896 shutdown_connections 2024-01-21T03:18:59.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 1 -- 172.21.15.7:0/4046902896 wait complete. 2024-01-21T03:18:59.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:18:59.190+0000 7f1151f41640 1 librados: shutdown 2024-01-21T03:18:59.187 INFO:tasks.ceph:check_enable_crimson: False 2024-01-21T03:18:59.187 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-21T03:18:59.188 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:59.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-21T03:18:59.215 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:59.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-21T03:18:59.229 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:18:59.242+0000 7f7ae9c13040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-21T03:18:59.255 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:59.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-21T03:18:59.286 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:18:59.299+0000 7f7ae9c13040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-21T03:18:59.314 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:18:59.314 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-21T03:18:59.377 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:18:59.391+0000 7f1ba1a43040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-21T03:18:59.381 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:59.381 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-21T03:18:59.411 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:59.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-21T03:18:59.436 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:18:59.450+0000 7f1ba1a43040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-21T03:18:59.462 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:59.462 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-21T03:18:59.528 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-01-21T03:18:59.529 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-21T03:18:59.596 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 9f4767aa-192e-4ec9-addf-a51c0872da7b 0 2024-01-21T03:18:59.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.762+0000 7fc0d1f2c640 10 monclient: get_monmap_and_config 2024-01-21T03:18:59.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.762+0000 7fc0d1f2c640 10 monclient: build_initial_monmap 2024-01-21T03:18:59.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.762+0000 7fc0d1f2c640 10 monclient: monmap: 2024-01-21T03:18:59.748 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:18:59.748 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:59.749 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:59.763094+0000 2024-01-21T03:18:59.749 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:59.763094+0000 2024-01-21T03:18:59.749 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:18:59.749 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:18:59.749 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:18:59.749 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:18:59.749 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:18:59.749 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:18:59.749 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:18:59.749 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:18:59.751 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.762+0000 7fc0d1f2c640 1 Processor -- start 2024-01-21T03:18:59.751 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.762+0000 7fc0d1f2c640 1 -- start start 2024-01-21T03:18:59.751 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.762+0000 7fc0d1f2c640 10 monclient: init 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 10 monclient: _reopen_session rank -1 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0cc0a8b20 addr v1:172.21.15.177:6789/0 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc0cc0a9d10 0x7fc0cc13f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 10 monclient(hunting): picked mon.noname-c con 0x7fc0cc0a9d10 addr v2:172.21.15.177:3300/0 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc0cc0b2310 0x7fc0cc13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 10 monclient(hunting): picked mon.noname-a con 0x7fc0cc0b2310 addr v2:172.21.15.7:3300/0 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0cc1456f0 con 0x7fc0cc0a8b20 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0cc0a98c0 con 0x7fc0cc0b2310 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0cc13fb00 con 0x7fc0cc0a9d10 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 10 monclient(hunting): _renew_subs 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.763+0000 7fc0d1f2c640 10 monclient(hunting): authenticate will time out at 1291.149639s 2024-01-21T03:18:59.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.764+0000 7fc0cbfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc0cc0b2310 0x7fc0cc13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.764+0000 7fc0caffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc0cc0a9d10 0x7fc0cc13f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.764+0000 7fc0cbfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc0cc0b2310 0x7fc0cc13f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.177:38824/0 (socket says 172.21.15.177:38824) 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.764+0000 7fc0cbfff640 1 -- 172.21.15.177:0/307972252 learned_addr learned my addr 172.21.15.177:0/307972252 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.764+0000 7fc0cbfff640 10 monclient(hunting): get_auth_request con 0x7fc0cc0b2310 auth_method 0 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.764+0000 7fc0cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.764+0000 7fc0cbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.764+0000 7fc0cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.764+0000 7fc0caffd640 10 monclient(hunting): get_auth_request con 0x7fc0cc0a9d10 auth_method 0 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.764+0000 7fc0caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.764+0000 7fc0caffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.764+0000 7fc0caffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.765+0000 7fc0cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.765+0000 7fc0cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.765+0000 7fc0cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.765+0000 7fc0caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:18:59.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.765+0000 7fc0caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:18:59.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.765+0000 7fc0caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:18:59.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.767+0000 7fc0cbfff640 10 monclient(hunting): handle_auth_done global_id 4107 payload 274 2024-01-21T03:18:59.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.767+0000 7fc0cbfff640 1 -- 172.21.15.177:0/307972252 >> v2:172.21.15.177:3300/0 conn(0x7fc0cc0a9d10 msgr2=0x7fc0cc13f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:18:59.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.767+0000 7fc0cbfff640 1 --2- 172.21.15.177:0/307972252 >> v2:172.21.15.177:3300/0 conn(0x7fc0cc0a9d10 0x7fc0cc13f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:59.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.767+0000 7fc0cbfff640 1 -- 172.21.15.177:0/307972252 >> v1:172.21.15.177:6789/0 conn(0x7fc0cc0a8b20 legacy=0x7fc0cc145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:18:59.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.767+0000 7fc0cbfff640 10 monclient: _finish_hunting 0 2024-01-21T03:18:59.755 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.767+0000 7fc0cbfff640 1 monclient: found mon.noname-a 2024-01-21T03:18:59.755 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.767+0000 7fc0cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:18:59.755 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.767+0000 7fc0cbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:18:59.755 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.767+0000 7fc0cbfff640 1 -- 172.21.15.177:0/307972252 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0cc146290 con 0x7fc0cc0b2310 2024-01-21T03:18:59.755 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.768+0000 7fc0cbfff640 1 --2- 172.21.15.177:0/307972252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0b2310 0x7fc0cc13f5c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0aa660 tx=0x7fc0c402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c02ad9d379e88e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:18:59.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0ca7fc640 1 -- 172.21.15.177:0/307972252 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c402c9f0 con 0x7fc0cc0b2310 2024-01-21T03:18:59.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:18:59.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:18:59.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0ca7fc640 10 monclient: dump: 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0ca7fc640 10 monclient: _finish_auth 0 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0ca7fc640 10 monclient: _check_auth_tickets 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0ca7fc640 1 -- 172.21.15.177:0/307972252 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c402cb50 con 0x7fc0cc0b2310 2024-01-21T03:18:59.760 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0ca7fc640 1 -- 172.21.15.177:0/307972252 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c40343f0 con 0x7fc0cc0b2310 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0d1f2c640 5 monclient: authenticate success, global_id 4107 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0d1f2c640 10 monclient: get_monmap_and_config success 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0d1f2c640 4 set_mon_vals no callback set 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.773+0000 7fc0ca7fc640 10 monclient: dump: 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:18:59.761 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:18:59.762 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:18:59.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.774+0000 7fc0d1f2c640 10 monclient: shutdown 2024-01-21T03:18:59.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.774+0000 7fc0d1f2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:18:59.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.774+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/307972252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0b2310 msgr2=0x7fc0cc13f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:18:59.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.774+0000 7fc0d1f2c640 1 --2- 172.21.15.177:0/307972252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0b2310 0x7fc0cc13f5c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0aa660 tx=0x7fc0c402be50 comp rx=0 tx=0).stop 2024-01-21T03:18:59.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.774+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/307972252 shutdown_connections 2024-01-21T03:18:59.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.774+0000 7fc0d1f2c640 1 --2- 172.21.15.177:0/307972252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0b2310 0x7fc0cc13f5c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:59.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.774+0000 7fc0d1f2c640 1 --2- 172.21.15.177:0/307972252 >> v2:172.21.15.177:3300/0 conn(0x7fc0cc0a9d10 0x7fc0cc13f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:59.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.774+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/307972252 >> 172.21.15.177:0/307972252 conn(0x7fc0cc066fb0 msgr2=0x7fc0cc0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:18:59.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.774+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/307972252 shutdown_connections 2024-01-21T03:18:59.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.774+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/307972252 wait complete. 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.774+0000 7fc0d1f2c640 10 monclient: build_initial_monmap 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.774+0000 7fc0d1f2c640 10 monclient: monmap: 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 1 librados: starting msgr at 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 1 librados: starting objecter 2024-01-21T03:18:59.766 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 1 Processor -- start 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 1 -- start start 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 1 librados: setting wanted keys 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 1 librados: calling monclient init 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 10 monclient: init 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 10 monclient: _reopen_session rank -1 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0a8b20 0x7fc0cc142830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 10 monclient(hunting): picked mon.noname-a con 0x7fc0cc0a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0cc0a9d10 0x7fc0cc13fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 10 monclient(hunting): picked mon.noname-b con 0x7fc0cc0a9d10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc0cc0b2310 0x7fc0cc1402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 10 monclient(hunting): picked mon.noname-c con 0x7fc0cc0b2310 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc1392c0 con 0x7fc0cc0a8b20 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:59.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc1395a0 con 0x7fc0cc0a9d10 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc143020 con 0x7fc0cc0b2310 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 10 monclient(hunting): _renew_subs 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.775+0000 7fc0d1f2c640 10 monclient(hunting): authenticate will time out at 1291.162060s 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0caffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0cc0a9d10 0x7fc0cc13fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0cb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0a8b20 0x7fc0cc142830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0cbfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc0cc0b2310 0x7fc0cc1402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0caffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0cc0a9d10 0x7fc0cc13fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:53166/0 (socket says 172.21.15.177:53166) 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0caffd640 1 -- 172.21.15.177:0/1979042242 learned_addr learned my addr 172.21.15.177:0/1979042242 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0caffd640 10 monclient(hunting): get_auth_request con 0x7fc0cc0a9d10 auth_method 0 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0caffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0caffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:18:59.768 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0cb7fe640 10 monclient(hunting): get_auth_request con 0x7fc0cc0a8b20 auth_method 0 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0cb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0cbfff640 10 monclient(hunting): get_auth_request con 0x7fc0cc0b2310 auth_method 0 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0cbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.776+0000 7fc0cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.777+0000 7fc0caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.777+0000 7fc0caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.777+0000 7fc0caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.777+0000 7fc0cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.777+0000 7fc0cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.777+0000 7fc0cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.777+0000 7fc0cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:18:59.769 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.777+0000 7fc0cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:18:59.770 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.777+0000 7fc0cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:18:59.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.779+0000 7fc0caffd640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-01-21T03:18:59.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.779+0000 7fc0caffd640 1 -- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc0cc0b2310 msgr2=0x7fc0cc1402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:18:59.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.779+0000 7fc0caffd640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc0cc0b2310 0x7fc0cc1402a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:59.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.779+0000 7fc0caffd640 1 -- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0a8b20 msgr2=0x7fc0cc142830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:18:59.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.779+0000 7fc0caffd640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0a8b20 0x7fc0cc142830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:59.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.779+0000 7fc0caffd640 10 monclient: _finish_hunting 0 2024-01-21T03:18:59.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.779+0000 7fc0caffd640 1 monclient: found mon.noname-b 2024-01-21T03:18:59.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.779+0000 7fc0caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:18:59.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.780+0000 7fc0caffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:18:59.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.780+0000 7fc0caffd640 1 -- 172.21.15.177:0/1979042242 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c4009b30 con 0x7fc0cc0a9d10 2024-01-21T03:18:59.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.782+0000 7fc0caffd640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0cc0a9d10 0x7fc0cc13fd60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc0c000e980 tx=0x7fc0c000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:18:59.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.782+0000 7fc0b3fff640 10 client.?.objecter ms_handle_connect 0x7fc0cc0a9d10 2024-01-21T03:18:59.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.782+0000 7fc0b3fff640 10 client.?.objecter resend_mon_ops 2024-01-21T03:18:59.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.785+0000 7fc0b3fff640 1 -- 172.21.15.177:0/1979042242 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0009670 con 0x7fc0cc0a9d10 2024-01-21T03:18:59.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:18:59.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:18:59.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0b3fff640 10 monclient: dump: 2024-01-21T03:18:59.775 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:18:59.775 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:59.775 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:59.775 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:59.775 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:18:59.775 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:18:59.775 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0b3fff640 10 monclient: _finish_auth 0 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0b3fff640 10 monclient: _check_auth_tickets 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0b3fff640 1 -- 172.21.15.177:0/1979042242 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c000ca00 con 0x7fc0cc0a9d10 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0d1f2c640 5 monclient: authenticate success, global_id 4114 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0d1f2c640 10 monclient: _renew_subs 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0d1f2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/1979042242 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0cc137ba0 con 0x7fc0cc0a9d10 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0b3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0b3fff640 1 -- 172.21.15.177:0/1979042242 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c000cd90 con 0x7fc0cc0a9d10 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:18:59.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0b3fff640 10 monclient: dump: 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0c9ffb640 4 set_mon_vals no callback set 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0d1f2c640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0d1f2c640 10 monclient: _renew_subs 2024-01-21T03:18:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0d1f2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/1979042242 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0cc142d70 con 0x7fc0cc0a9d10 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.786+0000 7fc0d1f2c640 1 librados: init done 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0d1f2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0d1f2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0d1f2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/1979042242 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0cc04e970 con 0x7fc0cc0a9d10 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0caffd640 1 -- 172.21.15.177:0/1979042242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0cc0a9d10 msgr2=0x7fc0cc13fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0caffd640 1 -- 172.21.15.177:0/1979042242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0cc0a9d10 msgr2=0x7fc0cc13fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0caffd640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0cc0a9d10 0x7fc0cc13fd60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc0c000e980 tx=0x7fc0c000ee40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0caffd640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0cc0a9d10 0x7fc0cc13fd60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc0c000e980 tx=0x7fc0c000ee40 comp rx=0 tx=0).stop 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 10 monclient: _reopen_session rank -1 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 1 -- 172.21.15.177:0/1979042242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0cc0a9d10 msgr2=0x7fc0cc13fd60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0cc0a9d10 0x7fc0cc13fd60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0a8b20 0x7fc0b8007200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 10 monclient(hunting): picked mon.a con 0x7fc0cc0a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0b8007bd0 0x7fc0b800a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 10 monclient(hunting): picked mon.b con 0x7fc0b8007bd0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc0cc0b2310 0x7fc0b800cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 10 monclient(hunting): picked mon.c con 0x7fc0cc0b2310 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 1 -- 172.21.15.177:0/1979042242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b800d590 con 0x7fc0cc0a8b20 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 1 -- 172.21.15.177:0/1979042242 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b800d6d0 con 0x7fc0b8007bd0 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 10 monclient(hunting): start opening mon connection 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 1 -- 172.21.15.177:0/1979042242 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b800d840 con 0x7fc0cc0b2310 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0b3fff640 10 monclient(hunting): _renew_subs 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.787+0000 7fc0cbfff640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc0cc0b2310 0x7fc0b800cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:59.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cb7fe640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0a8b20 0x7fc0b8007200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cbfff640 10 monclient(hunting): get_auth_request con 0x7fc0cc0b2310 auth_method 0 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cb7fe640 10 monclient(hunting): get_auth_request con 0x7fc0cc0a8b20 auth_method 0 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.788+0000 7fc0cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T03:18:59.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.789+0000 7fc0cb7fe640 1 -- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0a8b20 msgr2=0x7fc0b8007200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-01-21T03:18:59.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.789+0000 7fc0cb7fe640 1 -- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0a8b20 msgr2=0x7fc0b8007200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-21T03:18:59.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.789+0000 7fc0cb7fe640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0a8b20 0x7fc0b8007200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T03:18:59.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.789+0000 7fc0cb7fe640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0a8b20 0x7fc0b8007200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T03:18:59.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 10 monclient(hunting): handle_auth_done global_id 4114 payload 293 2024-01-21T03:18:59.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 1 -- 172.21.15.177:0/1979042242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0b8007bd0 msgr2=0x7fc0b800a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:18:59.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0b8007bd0 0x7fc0b800a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:59.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 1 -- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0a8b20 msgr2=0x7fc0b8007200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T03:18:59.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0a8b20 0x7fc0b8007200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:18:59.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 10 monclient: _finish_hunting 0 2024-01-21T03:18:59.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 1 monclient: found mon.c 2024-01-21T03:18:59.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:18:59.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:18:59.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 1 -- 172.21.15.177:0/1979042242 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fc0b800db30 con 0x7fc0cc0b2310 2024-01-21T03:18:59.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:18:59.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:18:59.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 1 -- 172.21.15.177:0/1979042242 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0cc146290 con 0x7fc0cc0b2310 2024-01-21T03:18:59.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 10 monclient: _finish_auth 0 2024-01-21T03:18:59.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 10 monclient: _check_auth_tickets 2024-01-21T03:18:59.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.790+0000 7fc0cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:18:59.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.794+0000 7fc0cbfff640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc0cc0b2310 0x7fc0b800cb80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc0c402c9c0 tx=0x7fc0c0009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:18:59.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.794+0000 7fc0b3fff640 10 client.4114.objecter ms_handle_connect 0x7fc0cc0b2310 2024-01-21T03:18:59.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.794+0000 7fc0b3fff640 10 client.4114.objecter resend_mon_ops 2024-01-21T03:18:59.936 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:18:59.949+0000 7f7ae9c13040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-21T03:18:59.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.972+0000 7fc0b3fff640 1 -- 172.21.15.177:0/1979042242 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4048470 con 0x7fc0cc0b2310 2024-01-21T03:18:59.961 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.972+0000 7fc0b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:18:59.961 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.972+0000 7fc0b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:18:59.961 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.972+0000 7fc0b3fff640 10 monclient: dump: 2024-01-21T03:18:59.961 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:18:59.961 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:18:59.962 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:59.962 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:18:59.962 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:18:59.962 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:18:59.962 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:18:59.962 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:18:59.963 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:18:59.963 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:18:59.963 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.972+0000 7fc0b3fff640 1 -- 172.21.15.177:0/1979042242 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c40485d0 con 0x7fc0cc0b2310 2024-01-21T03:18:59.963 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.972+0000 7fc0b3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:18:59.963 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.972+0000 7fc0c97fa640 4 set_mon_vals no callback set 2024-01-21T03:18:59.966 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.979+0000 7fc0b3fff640 1 -- 172.21.15.177:0/1979042242 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7fc0c4048b00 con 0x7fc0cc0b2310 2024-01-21T03:18:59.966 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.979+0000 7fc0b3fff640 1 -- 172.21.15.177:0/1979042242 <== mon.2 v2:172.21.15.7:3301/0 4 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc0c4034460 con 0x7fc0cc0b2310 2024-01-21T03:18:59.966 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.979+0000 7fc0b3fff640 10 client.4114.objecter ms_dispatch 0x7fc0cc000b90 osd_map(2..2 src has 1..2) v4 2024-01-21T03:18:59.966 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.979+0000 7fc0b3fff640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-01-21T03:18:59.966 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.979+0000 7fc0b3fff640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-01-21T03:18:59.966 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.979+0000 7fc0b3fff640 20 client.4114.objecter dump_active .. 0 homeless 2024-01-21T03:18:59.966 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.979+0000 7fc0b3fff640 1 -- 172.21.15.177:0/1979042242 <== mon.2 v2:172.21.15.7:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7fc0c4043070 con 0x7fc0cc0b2310 2024-01-21T03:18:59.966 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.979+0000 7fc0b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:18:59.966 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:18:59.979+0000 7fc0b3fff640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:18:59.991 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:00.004+0000 7f7ae9c13040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-21T03:18:59.998 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:00.012+0000 7f1ba1a43040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-21T03:19:00.037 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:00.051+0000 7f7ae9c13040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-21T03:19:00.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.051+0000 7fc0d1f2c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9f4767aa-192e-4ec9-addf-a51c0872da7b", "id": 0}] 2024-01-21T03:19:00.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.051+0000 7fc0d1f2c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9f4767aa-192e-4ec9-addf-a51c0872da7b", "id": 0}] 2024-01-21T03:19:00.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.051+0000 7fc0d1f2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:19:00.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.051+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/1979042242 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9f4767aa-192e-4ec9-addf-a51c0872da7b", "id": 0} v 0) v1 -- 0x7fc0cc14c600 con 0x7fc0cc0b2310 2024-01-21T03:19:00.051 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:00.065+0000 7f1ba1a43040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-21T03:19:00.093 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:00.107+0000 7f7ae9c13040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-21T03:19:00.095 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:00.109+0000 7f1ba1a43040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-21T03:19:00.137 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:00.151+0000 7f7ae9c13040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-21T03:19:00.147 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:00.161+0000 7f1ba1a43040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-21T03:19:00.192 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:00.205+0000 7f1ba1a43040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-21T03:19:00.245 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:00.259+0000 7f7ae9c13040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-21T03:19:00.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.261+0000 7fc0b3fff640 1 -- 172.21.15.177:0/1979042242 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9f4767aa-192e-4ec9-addf-a51c0872da7b", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc0c402c7f0 con 0x7fc0cc0b2310 2024-01-21T03:19:00.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.261+0000 7fc0b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9f4767aa-192e-4ec9-addf-a51c0872da7b", "id": 0}] 2024-01-21T03:19:00.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.261+0000 7fc0b3fff640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:00.247 INFO:teuthology.orchestra.run.smithi177.stdout:0 2024-01-21T03:19:00.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 10 librados: watch_flush enter 2024-01-21T03:19:00.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 10 librados: watch_flush exit 2024-01-21T03:19:00.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 20 client.4114.objecter shutdown clearing up homeless session... 2024-01-21T03:19:00.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 10 client.4114.objecter successfully canceled tick 2024-01-21T03:19:00.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 10 monclient: shutdown 2024-01-21T03:19:00.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:00.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc0cc0b2310 msgr2=0x7fc0b800cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:00.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc0cc0b2310 0x7fc0b800cb80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc0c402c9c0 tx=0x7fc0c0009190 comp rx=0 tx=0).stop 2024-01-21T03:19:00.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/1979042242 shutdown_connections 2024-01-21T03:19:00.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc0cc0b2310 0x7fc0b800cb80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0b8007bd0 0x7fc0b800a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 1 --2- 172.21.15.177:0/1979042242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0cc0a8b20 0x7fc0b8007200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/1979042242 >> 172.21.15.177:0/1979042242 conn(0x7fc0cc066fb0 msgr2=0x7fc0cc145b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:00.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/1979042242 shutdown_connections 2024-01-21T03:19:00.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 1 -- 172.21.15.177:0/1979042242 wait complete. 2024-01-21T03:19:00.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.262+0000 7fc0d1f2c640 1 librados: shutdown 2024-01-21T03:19:00.258 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 066d7378-561f-4e74-b97b-2ead2fa2625e 1 2024-01-21T03:19:00.318 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:00.332+0000 7f7ae9c13040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-21T03:19:00.323 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:00.337+0000 7f1ba1a43040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-21T03:19:00.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.371+0000 7f44a024f640 10 monclient: get_monmap_and_config 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.371+0000 7f44a024f640 10 monclient: build_initial_monmap 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.371+0000 7f44a024f640 10 monclient: monmap: 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:19:00.372096+0000 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:19:00.372096+0000 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:00.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.371+0000 7f44a024f640 1 Processor -- start 2024-01-21T03:19:00.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.371+0000 7f44a024f640 1 -- start start 2024-01-21T03:19:00.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.371+0000 7f44a024f640 10 monclient: init 2024-01-21T03:19:00.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.372+0000 7f44a024f640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:00.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.372+0000 7f44a024f640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-01-21T03:19:00.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.372+0000 7f44a024f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44980a8b20 0x7f4498145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:00.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.372+0000 7f44a024f640 10 monclient(hunting): picked mon.noname-a con 0x7f44980a8b20 addr v2:172.21.15.7:3300/0 2024-01-21T03:19:00.361 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.372+0000 7f44a024f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44980a9d10 addr v1:172.21.15.7:6789/0 2024-01-21T03:19:00.361 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.372+0000 7f44a024f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44980b2310 addr v1:172.21.15.177:6789/0 2024-01-21T03:19:00.361 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.372+0000 7f44a024f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44981456f0 con 0x7f44980a9d10 2024-01-21T03:19:00.361 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.372+0000 7f44a024f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44980a98c0 con 0x7f44980b2310 2024-01-21T03:19:00.361 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.372+0000 7f44a024f640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:00.361 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.372+0000 7f44a024f640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44981460c0 con 0x7f44980a8b20 2024-01-21T03:19:00.361 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.372+0000 7f44a024f640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:00.361 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.372+0000 7f44a024f640 10 monclient(hunting): authenticate will time out at 1291.758717s 2024-01-21T03:19:00.361 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.373+0000 7f449e7ec640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f44980b2310 0x7f449814c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:36752/0 (socket says 172.21.15.177:36752) 2024-01-21T03:19:00.361 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.373+0000 7f449e7ec640 1 -- 172.21.15.177:0/166591371 learned_addr learned my addr 172.21.15.177:0/166591371 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-01-21T03:19:00.361 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.373+0000 7f449dfeb640 1 --2- 172.21.15.177:0/166591371 >> v2:172.21.15.7:3300/0 conn(0x7f44980a8b20 0x7f4498145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:00.361 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.373+0000 7f449dfeb640 10 monclient(hunting): get_auth_request con 0x7f44980a8b20 auth_method 0 2024-01-21T03:19:00.363 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.373+0000 7f449dfeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:00.363 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.373+0000 7f449dfeb640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:00.363 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.373+0000 7f449dfeb640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:00.363 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.374+0000 7f449dfeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:00.363 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.374+0000 7f449dfeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:00.363 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.374+0000 7f449dfeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:00.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.376+0000 7f449dfeb640 10 monclient(hunting): handle_auth_done global_id 4113 payload 274 2024-01-21T03:19:00.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.376+0000 7f449dfeb640 1 -- 172.21.15.177:0/166591371 >> v1:172.21.15.177:6789/0 conn(0x7f44980b2310 legacy=0x7f449814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:00.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.376+0000 7f449dfeb640 1 -- 172.21.15.177:0/166591371 >> v1:172.21.15.7:6789/0 conn(0x7f44980a9d10 legacy=0x7f449813f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:00.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.376+0000 7f449dfeb640 10 monclient: _finish_hunting 0 2024-01-21T03:19:00.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.376+0000 7f449dfeb640 1 monclient: found mon.noname-a 2024-01-21T03:19:00.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.376+0000 7f449cfe9640 1 -- 172.21.15.177:0/166591371 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260728971 0 0) 0x7f44980a98c0 con 0x7f44980b2310 2024-01-21T03:19:00.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.376+0000 7f449dfeb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:00.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.376+0000 7f449dfeb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:19:00.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.376+0000 7f449dfeb640 1 -- 172.21.15.177:0/166591371 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4498146200 con 0x7f44980a8b20 2024-01-21T03:19:00.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.376+0000 7f449cfe9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-21T03:19:00.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.380+0000 7f449dfeb640 1 --2- 172.21.15.177:0/166591371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980a8b20 0x7f4498145b80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f44940097a0 tx=0x7f449402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a56c86465a2ad67 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:00.366 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:00.380+0000 7f7ae9c13040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-21T03:19:00.370 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f449cfe9640 1 -- 172.21.15.177:0/166591371 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449402e070 con 0x7f44980a8b20 2024-01-21T03:19:00.370 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f449cfe9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:00.370 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f449cfe9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:19:00.370 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f449cfe9640 10 monclient: dump: 2024-01-21T03:19:00.370 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:00.370 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:00.370 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.370 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.370 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:00.370 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f449cfe9640 10 monclient: _finish_auth 0 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f449cfe9640 10 monclient: _check_auth_tickets 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f449cfe9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f449cfe9640 1 -- 172.21.15.177:0/166591371 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44940315a0 con 0x7f44980a8b20 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f44a024f640 5 monclient: authenticate success, global_id 4113 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f44a024f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f449cfe9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f44a024f640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f449cfe9640 1 -- 172.21.15.177:0/166591371 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494031930 con 0x7f44980a8b20 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f449cfe9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:00.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f44a024f640 4 set_mon_vals no callback set 2024-01-21T03:19:00.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f449cfe9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:19:00.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f449cfe9640 10 monclient: dump: 2024-01-21T03:19:00.372 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:00.372 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:00.372 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.372 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.372 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:00.372 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:00.372 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:00.372 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:00.372 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:00.372 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:00.376 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f44a024f640 10 monclient: shutdown 2024-01-21T03:19:00.376 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f44a024f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:00.376 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f44a024f640 1 -- 172.21.15.177:0/166591371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980a8b20 msgr2=0x7f4498145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:00.376 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f44a024f640 1 --2- 172.21.15.177:0/166591371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980a8b20 0x7f4498145b80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f44940097a0 tx=0x7f449402b840 comp rx=0 tx=0).stop 2024-01-21T03:19:00.376 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f44a024f640 1 -- 172.21.15.177:0/166591371 shutdown_connections 2024-01-21T03:19:00.376 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f44a024f640 1 --2- 172.21.15.177:0/166591371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980a8b20 0x7f4498145b80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.376 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.384+0000 7f44a024f640 1 -- 172.21.15.177:0/166591371 >> 172.21.15.177:0/166591371 conn(0x7f4498066fb0 msgr2=0x7f44980b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:00.376 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.385+0000 7f44a024f640 1 -- 172.21.15.177:0/166591371 shutdown_connections 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.385+0000 7f44a024f640 1 -- 172.21.15.177:0/166591371 wait complete. 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.385+0000 7f44a024f640 10 monclient: build_initial_monmap 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.385+0000 7f44a024f640 10 monclient: monmap: 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.385+0000 7f44a024f640 1 librados: starting msgr at 2024-01-21T03:19:00.377 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.385+0000 7f44a024f640 1 librados: starting objecter 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.385+0000 7f44a024f640 1 Processor -- start 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 1 -- start start 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 1 librados: setting wanted keys 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 1 librados: calling monclient init 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 10 monclient: init 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980a8b20 0x7f4498143e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 10 monclient(hunting): picked mon.noname-a con 0x7f44980a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44980a9d10 0x7f4498140c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 10 monclient(hunting): picked mon.noname-b con 0x7f44980a9d10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44980b2310 0x7f44981411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 10 monclient(hunting): picked mon.noname-c con 0x7f44980b2310 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:00.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44981407d0 con 0x7f44980a8b20 2024-01-21T03:19:00.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:00.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4498144e20 con 0x7f44980a9d10 2024-01-21T03:19:00.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:00.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4498145100 con 0x7f44980b2310 2024-01-21T03:19:00.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:00.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.386+0000 7f44a024f640 10 monclient(hunting): authenticate will time out at 1291.772658s 2024-01-21T03:19:00.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449d7ea640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44980a9d10 0x7f4498140c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:00.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449e7ec640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44980b2310 0x7f44981411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:00.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449dfeb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980a8b20 0x7f4498143e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:00.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449d7ea640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44980a9d10 0x7f4498140c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:53180/0 (socket says 172.21.15.177:53180) 2024-01-21T03:19:00.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449d7ea640 1 -- 172.21.15.177:0/3873549390 learned_addr learned my addr 172.21.15.177:0/3873549390 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449d7ea640 10 monclient(hunting): get_auth_request con 0x7f44980a9d10 auth_method 0 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449d7ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449d7ea640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449d7ea640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449e7ec640 10 monclient(hunting): get_auth_request con 0x7f44980b2310 auth_method 0 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449e7ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449e7ec640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449e7ec640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449dfeb640 10 monclient(hunting): get_auth_request con 0x7f44980a8b20 auth_method 0 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449dfeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449dfeb640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.387+0000 7f449dfeb640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.388+0000 7f449d7ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:00.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.388+0000 7f449d7ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:00.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.388+0000 7f449d7ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:00.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.388+0000 7f449e7ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:00.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.388+0000 7f449e7ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:00.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.388+0000 7f449e7ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:00.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.388+0000 7f449dfeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:00.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.388+0000 7f449dfeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:00.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.388+0000 7f449dfeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:00.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.390+0000 7f449d7ea640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-01-21T03:19:00.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.390+0000 7f449d7ea640 1 -- 172.21.15.177:0/3873549390 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44980b2310 msgr2=0x7f44981411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:00.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.390+0000 7f449d7ea640 1 --2- 172.21.15.177:0/3873549390 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44980b2310 0x7f44981411a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.390+0000 7f449d7ea640 1 -- 172.21.15.177:0/3873549390 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980a8b20 msgr2=0x7f4498143e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:00.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.390+0000 7f449d7ea640 1 --2- 172.21.15.177:0/3873549390 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980a8b20 0x7f4498143e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.390+0000 7f449d7ea640 10 monclient: _finish_hunting 0 2024-01-21T03:19:00.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.390+0000 7f449d7ea640 1 monclient: found mon.noname-b 2024-01-21T03:19:00.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.390+0000 7f449d7ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:00.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.390+0000 7f449d7ea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:19:00.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.390+0000 7f449d7ea640 1 -- 172.21.15.177:0/3873549390 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4494009400 con 0x7f44980a9d10 2024-01-21T03:19:00.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.392+0000 7f449d7ea640 1 --2- 172.21.15.177:0/3873549390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44980a9d10 0x7f4498140c60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f448800c400 tx=0x7f448800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:00.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.392+0000 7f448e7fc640 10 client.?.objecter ms_handle_connect 0x7f44980a9d10 2024-01-21T03:19:00.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.392+0000 7f448e7fc640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:00.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448e7fc640 1 -- 172.21.15.177:0/3873549390 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4488011790 con 0x7f44980a9d10 2024-01-21T03:19:00.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:00.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:00.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448e7fc640 10 monclient: dump: 2024-01-21T03:19:00.386 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:00.386 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:00.386 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.386 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.386 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:00.386 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:00.386 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:00.386 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:00.386 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:00.386 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:00.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448e7fc640 10 monclient: _finish_auth 0 2024-01-21T03:19:00.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448e7fc640 10 monclient: _check_auth_tickets 2024-01-21T03:19:00.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:00.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448e7fc640 1 -- 172.21.15.177:0/3873549390 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4488011930 con 0x7f44980a9d10 2024-01-21T03:19:00.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f44a024f640 5 monclient: authenticate success, global_id 4120 2024-01-21T03:19:00.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f44a024f640 10 monclient: _renew_subs 2024-01-21T03:19:00.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f44a024f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:00.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f44a024f640 1 -- 172.21.15.177:0/3873549390 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4498144340 con 0x7f44980a9d10 2024-01-21T03:19:00.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:00.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448e7fc640 1 -- 172.21.15.177:0/3873549390 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4488011ce0 con 0x7f44980a9d10 2024-01-21T03:19:00.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:00.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:00.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448e7fc640 10 monclient: dump: 2024-01-21T03:19:00.388 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:00.388 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:00.388 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.388 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.388 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:00.388 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:00.388 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:00.388 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:00.389 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:00.389 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:00.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f448f7fe640 4 set_mon_vals no callback set 2024-01-21T03:19:00.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f44a024f640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:00.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f44a024f640 10 monclient: _renew_subs 2024-01-21T03:19:00.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f44a024f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:00.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f44a024f640 1 -- 172.21.15.177:0/3873549390 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4498136af0 con 0x7f44980a9d10 2024-01-21T03:19:00.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.397+0000 7f44a024f640 1 librados: init done 2024-01-21T03:19:00.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.398+0000 7f44a024f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:00.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.398+0000 7f44a024f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:00.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.398+0000 7f44a024f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:00.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.398+0000 7f44a024f640 1 -- 172.21.15.177:0/3873549390 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4498140980 con 0x7f44980a9d10 2024-01-21T03:19:00.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.400+0000 7f448e7fc640 1 -- 172.21.15.177:0/3873549390 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f448801c7e0 con 0x7f44980a9d10 2024-01-21T03:19:00.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.400+0000 7f448e7fc640 1 -- 172.21.15.177:0/3873549390 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f4488026b80 con 0x7f44980a9d10 2024-01-21T03:19:00.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.400+0000 7f448e7fc640 10 client.4120.objecter ms_dispatch 0x7f4498000b90 osd_map(3..3 src has 1..3) v4 2024-01-21T03:19:00.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.400+0000 7f448e7fc640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2024-01-21T03:19:00.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.400+0000 7f448e7fc640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2024-01-21T03:19:00.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.400+0000 7f448e7fc640 20 client.4120.objecter dump_active .. 0 homeless 2024-01-21T03:19:00.400 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:00.414+0000 7f1ba1a43040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-21T03:19:00.425 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:00.439+0000 7f7ae9c13040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-21T03:19:00.447 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:00.461+0000 7f1ba1a43040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-21T03:19:00.475 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:00.489+0000 7f7ae9c13040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-21T03:19:00.486 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.500+0000 7f448e7fc640 1 -- 172.21.15.177:0/3873549390 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f4498140980 con 0x7f44980a9d10 2024-01-21T03:19:00.486 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.500+0000 7f448e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:00.486 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.500+0000 7f448e7fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:00.504 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:00.519+0000 7f1ba1a43040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-21T03:19:00.553 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:00.567+0000 7f1ba1a43040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-21T03:19:00.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.572+0000 7f44a024f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "066d7378-561f-4e74-b97b-2ead2fa2625e", "id": 1}] 2024-01-21T03:19:00.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.572+0000 7f44a024f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "066d7378-561f-4e74-b97b-2ead2fa2625e", "id": 1}] 2024-01-21T03:19:00.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.572+0000 7f44a024f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:00.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.572+0000 7f44a024f640 1 -- 172.21.15.177:0/3873549390 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "066d7378-561f-4e74-b97b-2ead2fa2625e", "id": 1} v 0) v1 -- 0x7f44980a9110 con 0x7f44980a9d10 2024-01-21T03:19:00.687 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.701+0000 7f448e7fc640 1 -- 172.21.15.177:0/3873549390 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "066d7378-561f-4e74-b97b-2ead2fa2625e", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4488026d90 con 0x7f44980a9d10 2024-01-21T03:19:00.687 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.701+0000 7f448e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "066d7378-561f-4e74-b97b-2ead2fa2625e", "id": 1}] 2024-01-21T03:19:00.688 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.701+0000 7f448e7fc640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:00.688 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-01-21T03:19:00.689 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 10 librados: watch_flush enter 2024-01-21T03:19:00.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 10 librados: watch_flush exit 2024-01-21T03:19:00.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 20 client.4120.objecter shutdown clearing up homeless session... 2024-01-21T03:19:00.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 10 client.4120.objecter successfully canceled tick 2024-01-21T03:19:00.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 10 monclient: shutdown 2024-01-21T03:19:00.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:00.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 1 -- 172.21.15.177:0/3873549390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44980a9d10 msgr2=0x7f4498140c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:00.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 1 --2- 172.21.15.177:0/3873549390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44980a9d10 0x7f4498140c60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f448800c400 tx=0x7f448800c8c0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 1 -- 172.21.15.177:0/3873549390 shutdown_connections 2024-01-21T03:19:00.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 1 --2- 172.21.15.177:0/3873549390 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44980b2310 0x7f44981411a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 1 --2- 172.21.15.177:0/3873549390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44980a9d10 0x7f4498140c60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 1 --2- 172.21.15.177:0/3873549390 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980a8b20 0x7f4498143e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 1 -- 172.21.15.177:0/3873549390 >> 172.21.15.177:0/3873549390 conn(0x7f4498066fb0 msgr2=0x7f44980b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:00.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 1 -- 172.21.15.177:0/3873549390 shutdown_connections 2024-01-21T03:19:00.691 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 1 -- 172.21.15.177:0/3873549390 wait complete. 2024-01-21T03:19:00.691 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.702+0000 7f44a024f640 1 librados: shutdown 2024-01-21T03:19:00.699 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new f219649b-4d50-40a8-b74e-5a7f16b3c8c4 2 2024-01-21T03:19:00.799 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.811+0000 7f01b4861640 10 monclient: get_monmap_and_config 2024-01-21T03:19:00.799 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.811+0000 7f01b4861640 10 monclient: build_initial_monmap 2024-01-21T03:19:00.799 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.811+0000 7f01b4861640 10 monclient: monmap: 2024-01-21T03:19:00.799 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:00.799 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:00.799 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:19:00.813554+0000 2024-01-21T03:19:00.799 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:19:00.813554+0000 2024-01-21T03:19:00.799 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:00.799 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:00.799 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:00.799 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:00.800 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:00.800 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:00.800 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:00.800 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:00.800 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.812+0000 7f01b4861640 1 Processor -- start 2024-01-21T03:19:00.800 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.812+0000 7f01b4861640 1 -- start start 2024-01-21T03:19:00.800 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.812+0000 7f01b4861640 10 monclient: init 2024-01-21T03:19:00.802 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.812+0000 7f01b4861640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:00.802 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.812+0000 7f01b4861640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-01-21T03:19:00.802 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.812+0000 7f01b4861640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01b00a8b20 addr v1:172.21.15.7:6789/0 2024-01-21T03:19:00.802 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.812+0000 7f01b4861640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 0x7f01b013f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:00.802 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01b4861640 10 monclient(hunting): picked mon.noname-b con 0x7f01b00a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:00.802 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01b4861640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01b00b2310 0x7f01b013f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:00.802 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01b4861640 10 monclient(hunting): picked mon.noname-a con 0x7f01b00b2310 addr v2:172.21.15.7:3300/0 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01b4861640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b01456f0 con 0x7f01b00a8b20 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01b4861640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01b4861640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b00a98c0 con 0x7f01b00b2310 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01b4861640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01b4861640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b013fb00 con 0x7f01b00a9d10 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01b4861640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01b4861640 10 monclient(hunting): authenticate will time out at 1292.200175s 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01aed9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01b00b2310 0x7f01b013f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01add9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 0x7f01b013f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01aed9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01b00b2310 0x7f01b013f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.177:38886/0 (socket says 172.21.15.177:38886) 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01aed9d640 1 -- 172.21.15.177:0/1531354916 learned_addr learned my addr 172.21.15.177:0/1531354916 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01aed9d640 10 monclient(hunting): get_auth_request con 0x7f01b00b2310 auth_method 0 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01aed9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:00.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.813+0000 7f01aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:00.804 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.814+0000 7f01add9b640 10 monclient(hunting): get_auth_request con 0x7f01b00a9d10 auth_method 0 2024-01-21T03:19:00.804 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.814+0000 7f01add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:00.804 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.814+0000 7f01add9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:00.804 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.814+0000 7f01add9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:00.804 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.814+0000 7f01aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:00.804 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.814+0000 7f01aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:00.804 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.814+0000 7f01aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:00.804 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.814+0000 7f01add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:00.804 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.814+0000 7f01add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:00.804 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.814+0000 7f01add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:00.805 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.816+0000 7f01add9b640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2024-01-21T03:19:00.805 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.816+0000 7f01add9b640 1 -- 172.21.15.177:0/1531354916 >> v2:172.21.15.7:3300/0 conn(0x7f01b00b2310 msgr2=0x7f01b013f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:00.805 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.816+0000 7f01add9b640 1 --2- 172.21.15.177:0/1531354916 >> v2:172.21.15.7:3300/0 conn(0x7f01b00b2310 0x7f01b013f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.805 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.816+0000 7f01add9b640 1 -- 172.21.15.177:0/1531354916 >> v1:172.21.15.7:6789/0 conn(0x7f01b00a8b20 legacy=0x7f01b0145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:00.805 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.816+0000 7f01add9b640 10 monclient: _finish_hunting 0 2024-01-21T03:19:00.805 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.816+0000 7f01add9b640 1 monclient: found mon.noname-b 2024-01-21T03:19:00.805 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.816+0000 7f01add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:00.805 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.816+0000 7f01add9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:19:00.805 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.816+0000 7f01add9b640 1 -- 172.21.15.177:0/1531354916 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b0146290 con 0x7f01b00a9d10 2024-01-21T03:19:00.806 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.818+0000 7f01add9b640 1 --2- 172.21.15.177:0/1531354916 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 0x7f01b013f080 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f01a40097a0 tx=0x7f01a402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e42222936998d99 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:00.809 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01ad59a640 1 -- 172.21.15.177:0/1531354916 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a402e070 con 0x7f01b00a9d10 2024-01-21T03:19:00.809 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:00.809 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01ad59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:00.809 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01ad59a640 10 monclient: dump: 2024-01-21T03:19:00.809 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:00.809 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:00.809 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01ad59a640 10 monclient: _finish_auth 0 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01ad59a640 10 monclient: _check_auth_tickets 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01ad59a640 1 -- 172.21.15.177:0/1531354916 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a40315a0 con 0x7f01b00a9d10 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01ad59a640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01ad59a640 1 -- 172.21.15.177:0/1531354916 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a4031930 con 0x7f01b00a9d10 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01b4861640 5 monclient: authenticate success, global_id 4112 2024-01-21T03:19:00.810 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01b4861640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01b4861640 4 set_mon_vals no callback set 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01ad59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01ad59a640 10 monclient: dump: 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:00.811 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:00.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01b4861640 10 monclient: shutdown 2024-01-21T03:19:00.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01b4861640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:00.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01b4861640 1 -- 172.21.15.177:0/1531354916 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 msgr2=0x7f01b013f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:00.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.822+0000 7f01b4861640 1 --2- 172.21.15.177:0/1531354916 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 0x7f01b013f080 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f01a40097a0 tx=0x7f01a402b840 comp rx=0 tx=0).stop 2024-01-21T03:19:00.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.823+0000 7f01b4861640 1 -- 172.21.15.177:0/1531354916 shutdown_connections 2024-01-21T03:19:00.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.823+0000 7f01b4861640 1 --2- 172.21.15.177:0/1531354916 >> v2:172.21.15.7:3300/0 conn(0x7f01b00b2310 0x7f01b013f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.823+0000 7f01b4861640 1 --2- 172.21.15.177:0/1531354916 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 0x7f01b013f080 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.823+0000 7f01b4861640 1 -- 172.21.15.177:0/1531354916 >> 172.21.15.177:0/1531354916 conn(0x7f01b0066fb0 msgr2=0x7f01b00b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.823+0000 7f01b4861640 1 -- 172.21.15.177:0/1531354916 shutdown_connections 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.823+0000 7f01b4861640 1 -- 172.21.15.177:0/1531354916 wait complete. 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.823+0000 7f01b4861640 10 monclient: build_initial_monmap 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.823+0000 7f01b4861640 10 monclient: monmap: 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:00.816 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 1 librados: starting msgr at 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 1 librados: starting objecter 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 1 Processor -- start 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 1 -- start start 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 1 librados: setting wanted keys 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 1 librados: calling monclient init 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 10 monclient: init 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01b00a8b20 0x7f01b01409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 10 monclient(hunting): picked mon.noname-a con 0x7f01b00a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 0x7f01b0140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 10 monclient(hunting): picked mon.noname-c con 0x7f01b00a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01b00b2310 0x7f01b013a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 10 monclient(hunting): picked mon.noname-b con 0x7f01b00b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b01416c0 con 0x7f01b00a8b20 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b0142b50 con 0x7f01b00b2310 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b013fd60 con 0x7f01b00a9d10 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.824+0000 7f01b4861640 10 monclient(hunting): authenticate will time out at 1292.211985s 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01ae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01b00a8b20 0x7f01b01409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01add9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 0x7f01b0140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01aed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01b00b2310 0x7f01b013a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01ae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01b00a8b20 0x7f01b01409d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.177:38900/0 (socket says 172.21.15.177:38900) 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01ae59c640 1 -- 172.21.15.177:0/62985665 learned_addr learned my addr 172.21.15.177:0/62985665 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01ae59c640 10 monclient(hunting): get_auth_request con 0x7f01b00a8b20 auth_method 0 2024-01-21T03:19:00.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01ae59c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01aed9d640 10 monclient(hunting): get_auth_request con 0x7f01b00b2310 auth_method 0 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01aed9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01add9b640 10 monclient(hunting): get_auth_request con 0x7f01b00a9d10 auth_method 0 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01add9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.825+0000 7f01add9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.826+0000 7f01aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.826+0000 7f01aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.826+0000 7f01aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.826+0000 7f01ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.826+0000 7f01ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:00.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.826+0000 7f01ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:00.820 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.826+0000 7f01add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:00.820 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.826+0000 7f01add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:00.820 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.826+0000 7f01add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:00.820 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.827+0000 7f01add9b640 1 -- 172.21.15.177:0/62985665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 msgr2=0x7f01b0140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-01-21T03:19:00.820 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.827+0000 7f01add9b640 1 -- 172.21.15.177:0/62985665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 msgr2=0x7f01b0140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-21T03:19:00.820 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.827+0000 7f01add9b640 1 --2- 172.21.15.177:0/62985665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 0x7f01b0140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T03:19:00.820 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.827+0000 7f01add9b640 1 --2- 172.21.15.177:0/62985665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 0x7f01b0140f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T03:19:00.821 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.829+0000 7f01aed9d640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-01-21T03:19:00.821 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.829+0000 7f01aed9d640 1 -- 172.21.15.177:0/62985665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 msgr2=0x7f01b0140f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T03:19:00.821 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.829+0000 7f01aed9d640 1 --2- 172.21.15.177:0/62985665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 0x7f01b0140f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.821 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.829+0000 7f01aed9d640 1 -- 172.21.15.177:0/62985665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01b00a8b20 msgr2=0x7f01b01409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:00.821 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.829+0000 7f01aed9d640 1 --2- 172.21.15.177:0/62985665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01b00a8b20 0x7f01b01409d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:00.821 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.829+0000 7f01aed9d640 10 monclient: _finish_hunting 0 2024-01-21T03:19:00.821 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.829+0000 7f01aed9d640 1 monclient: found mon.noname-b 2024-01-21T03:19:00.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.829+0000 7f01aed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:00.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.829+0000 7f01aed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:19:00.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.829+0000 7f01aed9d640 1 -- 172.21.15.177:0/62985665 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a4009400 con 0x7f01b00b2310 2024-01-21T03:19:00.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.831+0000 7f01aed9d640 1 --2- 172.21.15.177:0/62985665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01b00b2310 0x7f01b013a090 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f01a800ef90 tx=0x7f01a800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:00.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.831+0000 7f019effd640 10 client.?.objecter ms_handle_connect 0x7f01b00b2310 2024-01-21T03:19:00.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.831+0000 7f019effd640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:00.823 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f019effd640 1 -- 172.21.15.177:0/62985665 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a800cd20 con 0x7f01b00b2310 2024-01-21T03:19:00.823 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f019effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:00.823 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f019effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:00.823 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f019effd640 10 monclient: dump: 2024-01-21T03:19:00.823 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:00.823 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:00.823 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.823 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.823 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:00.823 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f019effd640 10 monclient: _finish_auth 0 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f019effd640 10 monclient: _check_auth_tickets 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f019effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f019effd640 1 -- 172.21.15.177:0/62985665 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a800f040 con 0x7f01b00b2310 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f019effd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f019effd640 1 -- 172.21.15.177:0/62985665 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a8003bd0 con 0x7f01b00b2310 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f01b4861640 5 monclient: authenticate success, global_id 4123 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f01b4861640 10 monclient: _renew_subs 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f01b4861640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f01b4861640 1 -- 172.21.15.177:0/62985665 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b0141b50 con 0x7f01b00b2310 2024-01-21T03:19:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f01acd99640 4 set_mon_vals no callback set 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f019effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f019effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f019effd640 10 monclient: dump: 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f01b4861640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f01b4861640 10 monclient: _renew_subs 2024-01-21T03:19:00.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f01b4861640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:00.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f01b4861640 1 -- 172.21.15.177:0/62985665 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b013a7b0 con 0x7f01b00b2310 2024-01-21T03:19:00.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.835+0000 7f01b4861640 1 librados: init done 2024-01-21T03:19:00.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.837+0000 7f01b4861640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:00.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.837+0000 7f01b4861640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:00.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.837+0000 7f01b4861640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:00.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.837+0000 7f01b4861640 1 -- 172.21.15.177:0/62985665 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b013aa00 con 0x7f01b00b2310 2024-01-21T03:19:00.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.839+0000 7f019effd640 1 -- 172.21.15.177:0/62985665 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f01a80043b0 con 0x7f01b00b2310 2024-01-21T03:19:00.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.839+0000 7f019effd640 1 -- 172.21.15.177:0/62985665 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f01a8014cc0 con 0x7f01b00b2310 2024-01-21T03:19:00.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.839+0000 7f019effd640 10 client.4123.objecter ms_dispatch 0x7f01b0000b90 osd_map(4..4 src has 1..4) v4 2024-01-21T03:19:00.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.839+0000 7f019effd640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-01-21T03:19:00.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.839+0000 7f019effd640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-01-21T03:19:00.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.839+0000 7f019effd640 20 client.4123.objecter dump_active .. 0 homeless 2024-01-21T03:19:00.885 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:00.899+0000 7f7ae9c13040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-21T03:19:00.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.936+0000 7f019effd640 1 -- 172.21.15.177:0/62985665 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f01a8014440 con 0x7f01b00b2310 2024-01-21T03:19:00.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.937+0000 7f019effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:00.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:00.937+0000 7f019effd640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:00.993 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:01.006+0000 7f1ba1a43040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-21T03:19:00.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.009+0000 7f01b4861640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f219649b-4d50-40a8-b74e-5a7f16b3c8c4", "id": 2}] 2024-01-21T03:19:00.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.009+0000 7f01b4861640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f219649b-4d50-40a8-b74e-5a7f16b3c8c4", "id": 2}] 2024-01-21T03:19:00.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.009+0000 7f01b4861640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:00.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.009+0000 7f01b4861640 1 -- 172.21.15.177:0/62985665 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f219649b-4d50-40a8-b74e-5a7f16b3c8c4", "id": 2} v 0) v1 -- 0x7f01b01414a0 con 0x7f01b00b2310 2024-01-21T03:19:01.026 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:01.040+0000 7f7ae9c13040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-21T03:19:01.093 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.106+0000 7f019effd640 1 -- 172.21.15.177:0/62985665 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f219649b-4d50-40a8-b74e-5a7f16b3c8c4", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f01a802c0b0 con 0x7f01b00b2310 2024-01-21T03:19:01.093 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.106+0000 7f019effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f219649b-4d50-40a8-b74e-5a7f16b3c8c4", "id": 2}] 2024-01-21T03:19:01.093 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.106+0000 7f019effd640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:01.093 INFO:teuthology.orchestra.run.smithi177.stdout:2 2024-01-21T03:19:01.095 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 10 librados: watch_flush enter 2024-01-21T03:19:01.095 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 10 librados: watch_flush exit 2024-01-21T03:19:01.095 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 20 client.4123.objecter shutdown clearing up homeless session... 2024-01-21T03:19:01.095 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 10 client.4123.objecter successfully canceled tick 2024-01-21T03:19:01.096 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 10 monclient: shutdown 2024-01-21T03:19:01.096 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:01.096 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 1 -- 172.21.15.177:0/62985665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01b00b2310 msgr2=0x7f01b013a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.096 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 1 --2- 172.21.15.177:0/62985665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01b00b2310 0x7f01b013a090 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f01a800ef90 tx=0x7f01a800c490 comp rx=0 tx=0).stop 2024-01-21T03:19:01.096 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 1 -- 172.21.15.177:0/62985665 shutdown_connections 2024-01-21T03:19:01.096 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 1 --2- 172.21.15.177:0/62985665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01b00b2310 0x7f01b013a090 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.096 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 1 --2- 172.21.15.177:0/62985665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f01b00a9d10 0x7f01b0140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.096 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 1 --2- 172.21.15.177:0/62985665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01b00a8b20 0x7f01b01409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.096 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 1 -- 172.21.15.177:0/62985665 >> 172.21.15.177:0/62985665 conn(0x7f01b0066fb0 msgr2=0x7f01b013ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:01.096 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 1 -- 172.21.15.177:0/62985665 shutdown_connections 2024-01-21T03:19:01.096 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 1 -- 172.21.15.177:0/62985665 wait complete. 2024-01-21T03:19:01.096 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.107+0000 7f01b4861640 1 librados: shutdown 2024-01-21T03:19:01.105 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 6379a402-760a-4f07-9491-0918752a18fb 3 2024-01-21T03:19:01.133 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:01.146+0000 7f1ba1a43040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-21T03:19:01.141 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:01.155+0000 7f7ae9c13040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-21T03:19:01.188 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:01.202+0000 7f7ae9c13040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-21T03:19:01.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.216+0000 7f697847c640 10 monclient: get_monmap_and_config 2024-01-21T03:19:01.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.216+0000 7f697847c640 10 monclient: build_initial_monmap 2024-01-21T03:19:01.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.216+0000 7f697847c640 10 monclient: monmap: 2024-01-21T03:19:01.204 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:01.204 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:01.204 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:19:01.218842+0000 2024-01-21T03:19:01.204 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:19:01.218842+0000 2024-01-21T03:19:01.204 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:01.204 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.204 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:01.204 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:01.204 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:01.205 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:01.205 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:01.205 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.206 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.217+0000 7f697847c640 1 Processor -- start 2024-01-21T03:19:01.206 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.217+0000 7f697847c640 1 -- start start 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.217+0000 7f697847c640 10 monclient: init 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.218+0000 7f697847c640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.218+0000 7f697847c640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.218+0000 7f697847c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69700a8b20 addr v1:172.21.15.7:6789/0 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.218+0000 7f697847c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f69700a9d10 0x7f697013f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.218+0000 7f697847c640 10 monclient(hunting): picked mon.noname-b con 0x7f69700a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.218+0000 7f697847c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69700b2310 addr v1:172.21.15.177:6789/0 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.218+0000 7f697847c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69701456f0 con 0x7f69700a8b20 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.218+0000 7f697847c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69700a98c0 con 0x7f69700b2310 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.218+0000 7f697847c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.218+0000 7f697847c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6970146290 con 0x7f69700a9d10 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.218+0000 7f697847c640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.218+0000 7f697847c640 10 monclient(hunting): authenticate will time out at 1292.605473s 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.219+0000 7f6976a19640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f69700b2310 0x7f697013f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:36754/0 (socket says 172.21.15.177:36754) 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.219+0000 7f6976a19640 1 -- 172.21.15.177:0/515432075 learned_addr learned my addr 172.21.15.177:0/515432075 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-01-21T03:19:01.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.219+0000 7f6975a17640 1 --2- 172.21.15.177:0/515432075 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f69700a9d10 0x7f697013f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:01.208 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.219+0000 7f6975a17640 10 monclient(hunting): get_auth_request con 0x7f69700a9d10 auth_method 0 2024-01-21T03:19:01.208 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.219+0000 7f6975a17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:01.208 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.219+0000 7f6975a17640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.208 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.219+0000 7f6975a17640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.220+0000 7f6975a17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:01.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.220+0000 7f6975a17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:01.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.220+0000 7f6975a17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:01.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975216640 1 -- 172.21.15.177:0/515432075 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968828027 0 0) 0x7f69700a98c0 con 0x7f69700b2310 2024-01-21T03:19:01.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975216640 10 monclient(hunting): _negotiate 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975216640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975216640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975216640 10 monclient(hunting): my global_id is 4126 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975216640 1 -- 172.21.15.177:0/515432075 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f695c003520 con 0x7f69700b2310 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975216640 1 -- 172.21.15.177:0/515432075 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668574170 0 0) 0x7f69701456f0 con 0x7f69700a8b20 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975216640 10 monclient(hunting): _negotiate 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975216640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975216640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975216640 10 monclient(hunting): my global_id is 4125 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975216640 1 -- 172.21.15.177:0/515432075 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69700a98c0 con 0x7f69700a8b20 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975a17640 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975a17640 1 -- 172.21.15.177:0/515432075 >> v1:172.21.15.177:6789/0 conn(0x7f69700b2310 legacy=0x7f697013f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.221+0000 7f6975a17640 1 -- 172.21.15.177:0/515432075 >> v1:172.21.15.7:6789/0 conn(0x7f69700a8b20 legacy=0x7f6970145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.222+0000 7f6975a17640 10 monclient: _finish_hunting 0 2024-01-21T03:19:01.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.222+0000 7f6975a17640 1 monclient: found mon.noname-b 2024-01-21T03:19:01.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.222+0000 7f6975a17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:01.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.222+0000 7f6975a17640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:19:01.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.222+0000 7f6975a17640 1 -- 172.21.15.177:0/515432075 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69701463d0 con 0x7f69700a9d10 2024-01-21T03:19:01.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.223+0000 7f6975a17640 1 --2- 172.21.15.177:0/515432075 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f69700a9d10 0x7f697013f080 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f696c0097d0 tx=0x7f696c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8be62195b3b790c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:01.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f6975216640 1 -- 172.21.15.177:0/515432075 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c031800 con 0x7f69700a9d10 2024-01-21T03:19:01.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f6975216640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:01.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f6975216640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:01.215 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f6975216640 10 monclient: dump: 2024-01-21T03:19:01.215 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:01.215 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:01.215 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.215 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.215 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:01.215 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.216 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:01.216 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:01.216 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:01.216 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.216 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f6975216640 10 monclient: _finish_auth 0 2024-01-21T03:19:01.216 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f6975216640 10 monclient: _check_auth_tickets 2024-01-21T03:19:01.216 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f6975216640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:01.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f6975216640 1 -- 172.21.15.177:0/515432075 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696c031960 con 0x7f69700a9d10 2024-01-21T03:19:01.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f697847c640 5 monclient: authenticate success, global_id 4118 2024-01-21T03:19:01.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f6975216640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:01.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f6975216640 1 -- 172.21.15.177:0/515432075 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c031cf0 con 0x7f69700a9d10 2024-01-21T03:19:01.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f697847c640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:01.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f697847c640 4 set_mon_vals no callback set 2024-01-21T03:19:01.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f6975216640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:01.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f6975216640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:19:01.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.227+0000 7f6975216640 10 monclient: dump: 2024-01-21T03:19:01.218 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:01.218 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:01.218 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.218 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.219 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:01.219 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.219 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:01.219 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:01.219 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:01.219 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.228+0000 7f697847c640 10 monclient: shutdown 2024-01-21T03:19:01.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.228+0000 7f697847c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:01.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.228+0000 7f697847c640 1 -- 172.21.15.177:0/515432075 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f69700a9d10 msgr2=0x7f697013f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.228+0000 7f697847c640 1 --2- 172.21.15.177:0/515432075 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f69700a9d10 0x7f697013f080 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f696c0097d0 tx=0x7f696c02b8a0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.228+0000 7f697847c640 1 -- 172.21.15.177:0/515432075 shutdown_connections 2024-01-21T03:19:01.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.228+0000 7f697847c640 1 --2- 172.21.15.177:0/515432075 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f69700a9d10 0x7f697013f080 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.231 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.228+0000 7f697847c640 1 -- 172.21.15.177:0/515432075 >> 172.21.15.177:0/515432075 conn(0x7f6970066fb0 msgr2=0x7f69700b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:01.231 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.228+0000 7f697847c640 1 -- 172.21.15.177:0/515432075 shutdown_connections 2024-01-21T03:19:01.231 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.228+0000 7f697847c640 1 -- 172.21.15.177:0/515432075 wait complete. 2024-01-21T03:19:01.231 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.229+0000 7f697847c640 10 monclient: build_initial_monmap 2024-01-21T03:19:01.231 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.229+0000 7f697847c640 10 monclient: monmap: 2024-01-21T03:19:01.231 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:01.231 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:01.232 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-01-21T03:19:01.232 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-01-21T03:19:01.232 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:01.232 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.232 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:01.232 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:01.233 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:01.233 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.233 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.229+0000 7f697847c640 1 librados: starting msgr at 2024-01-21T03:19:01.233 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.229+0000 7f697847c640 1 librados: starting objecter 2024-01-21T03:19:01.233 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.229+0000 7f697847c640 1 Processor -- start 2024-01-21T03:19:01.233 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.229+0000 7f697847c640 1 -- start start 2024-01-21T03:19:01.233 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.229+0000 7f697847c640 1 librados: setting wanted keys 2024-01-21T03:19:01.234 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.229+0000 7f697847c640 1 librados: calling monclient init 2024-01-21T03:19:01.234 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.229+0000 7f697847c640 10 monclient: init 2024-01-21T03:19:01.234 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:01.234 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:19:01.234 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69700a8b20 0x7f6970144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:01.234 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 10 monclient(hunting): picked mon.noname-a con 0x7f69700a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:01.234 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f69700a9d10 0x7f6970137540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:01.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 10 monclient(hunting): picked mon.noname-c con 0x7f69700a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:01.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69700b2310 0x7f6970136c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:01.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 10 monclient(hunting): picked mon.noname-b con 0x7f69700b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:01.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:01.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6970143400 con 0x7f69700a8b20 2024-01-21T03:19:01.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:01.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69701384c0 con 0x7f69700b2310 2024-01-21T03:19:01.236 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:01.236 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6970138730 con 0x7f69700a9d10 2024-01-21T03:19:01.236 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:01.236 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f697847c640 10 monclient(hunting): authenticate will time out at 1292.617509s 2024-01-21T03:19:01.236 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f6976a19640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69700b2310 0x7f6970136c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:01.236 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.230+0000 7f6975a17640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f69700a9d10 0x7f6970137540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:01.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976218640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69700a8b20 0x7f6970144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:01.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976a19640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69700b2310 0x7f6970136c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:53208/0 (socket says 172.21.15.177:53208) 2024-01-21T03:19:01.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976a19640 1 -- 172.21.15.177:0/2163666717 learned_addr learned my addr 172.21.15.177:0/2163666717 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-01-21T03:19:01.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976218640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69700a8b20 0x7f6970144e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.177:38908/0 (socket says 172.21.15.177:38908) 2024-01-21T03:19:01.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976a19640 10 monclient(hunting): get_auth_request con 0x7f69700b2310 auth_method 0 2024-01-21T03:19:01.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976a19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:01.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976a19640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976a19640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976218640 10 monclient(hunting): get_auth_request con 0x7f69700a8b20 auth_method 0 2024-01-21T03:19:01.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976218640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:01.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976218640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976218640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6975a17640 10 monclient(hunting): get_auth_request con 0x7f69700a9d10 auth_method 0 2024-01-21T03:19:01.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6975a17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:01.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6975a17640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6975a17640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976a19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:01.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976a19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:01.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976a19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:01.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976218640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:01.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976218640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:01.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6976218640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:01.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6975a17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:01.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6975a17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:01.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.231+0000 7f6975a17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:01.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.234+0000 7f6976a19640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-01-21T03:19:01.241 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.234+0000 7f6976a19640 1 -- 172.21.15.177:0/2163666717 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f69700a9d10 msgr2=0x7f6970137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.241 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.234+0000 7f6976a19640 1 --2- 172.21.15.177:0/2163666717 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f69700a9d10 0x7f6970137540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.241 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.234+0000 7f6976a19640 1 -- 172.21.15.177:0/2163666717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69700a8b20 msgr2=0x7f6970144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.241 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.234+0000 7f6976a19640 1 --2- 172.21.15.177:0/2163666717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69700a8b20 0x7f6970144e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.241 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.234+0000 7f6976a19640 10 monclient: _finish_hunting 0 2024-01-21T03:19:01.241 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.234+0000 7f6976a19640 1 monclient: found mon.noname-b 2024-01-21T03:19:01.242 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.234+0000 7f6976a19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:01.242 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.234+0000 7f6976a19640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:19:01.242 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.234+0000 7f6976a19640 1 -- 172.21.15.177:0/2163666717 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c009400 con 0x7f69700b2310 2024-01-21T03:19:01.242 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.236+0000 7f6976a19640 1 --2- 172.21.15.177:0/2163666717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69700b2310 0x7f6970136c30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f696800ca70 tx=0x7f696800cf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:01.242 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.236+0000 7f6966ffd640 10 client.?.objecter ms_handle_connect 0x7f69700b2310 2024-01-21T03:19:01.242 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.236+0000 7f6966ffd640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:01.242 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6966ffd640 1 -- 172.21.15.177:0/2163666717 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696800abd0 con 0x7f69700b2310 2024-01-21T03:19:01.242 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:01.243 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6966ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:01.243 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6966ffd640 10 monclient: dump: 2024-01-21T03:19:01.243 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:01.243 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:01.243 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.243 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.243 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:01.244 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.244 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:01.244 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:01.244 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:01.244 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.244 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6966ffd640 10 monclient: _finish_auth 0 2024-01-21T03:19:01.245 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6966ffd640 10 monclient: _check_auth_tickets 2024-01-21T03:19:01.245 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:01.245 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6966ffd640 1 -- 172.21.15.177:0/2163666717 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696800ad50 con 0x7f69700b2310 2024-01-21T03:19:01.245 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f697847c640 5 monclient: authenticate success, global_id 4129 2024-01-21T03:19:01.245 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f697847c640 10 monclient: _renew_subs 2024-01-21T03:19:01.245 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f697847c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:01.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f697847c640 1 -- 172.21.15.177:0/2163666717 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6970138a10 con 0x7f69700b2310 2024-01-21T03:19:01.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6966ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:01.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6966ffd640 1 -- 172.21.15.177:0/2163666717 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968012640 con 0x7f69700b2310 2024-01-21T03:19:01.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:01.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6966ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:01.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6966ffd640 10 monclient: dump: 2024-01-21T03:19:01.246 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:01.247 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:01.247 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.247 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.247 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:01.247 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.247 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:01.248 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:01.248 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:01.248 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.248 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f6974a15640 4 set_mon_vals no callback set 2024-01-21T03:19:01.248 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f697847c640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:01.248 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f697847c640 10 monclient: _renew_subs 2024-01-21T03:19:01.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f697847c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:01.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f697847c640 1 -- 172.21.15.177:0/2163666717 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6970137170 con 0x7f69700b2310 2024-01-21T03:19:01.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f697847c640 1 librados: init done 2024-01-21T03:19:01.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.240+0000 7f697847c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:01.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.241+0000 7f697847c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:01.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.241+0000 7f697847c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:01.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.241+0000 7f697847c640 1 -- 172.21.15.177:0/2163666717 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69701435b0 con 0x7f69700b2310 2024-01-21T03:19:01.250 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:01.259+0000 7f1ba1a43040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-21T03:19:01.292 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:01.306+0000 7f7ae9c13040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-21T03:19:01.293 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:01.306+0000 7f1ba1a43040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-21T03:19:01.328 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.340+0000 7f6966ffd640 1 -- 172.21.15.177:0/2163666717 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f6968012b20 con 0x7f69700b2310 2024-01-21T03:19:01.328 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.341+0000 7f6966ffd640 1 -- 172.21.15.177:0/2163666717 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f6968007500 con 0x7f69700b2310 2024-01-21T03:19:01.328 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.341+0000 7f6966ffd640 10 client.4129.objecter ms_dispatch 0x7f6970000b90 osd_map(5..5 src has 1..5) v4 2024-01-21T03:19:01.328 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.341+0000 7f6966ffd640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-01-21T03:19:01.329 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.341+0000 7f6966ffd640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-01-21T03:19:01.329 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.341+0000 7f6966ffd640 20 client.4129.objecter dump_active .. 0 homeless 2024-01-21T03:19:01.329 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.341+0000 7f6966ffd640 1 -- 172.21.15.177:0/2163666717 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f69701435b0 con 0x7f69700b2310 2024-01-21T03:19:01.329 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.341+0000 7f6966ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:01.329 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.341+0000 7f6966ffd640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:01.392 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:01.404+0000 7f7ae9c13040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-21T03:19:01.394 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:01.407+0000 7f1ba1a43040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-21T03:19:01.400 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.412+0000 7f697847c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6379a402-760a-4f07-9491-0918752a18fb", "id": 3}] 2024-01-21T03:19:01.400 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.412+0000 7f697847c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6379a402-760a-4f07-9491-0918752a18fb", "id": 3}] 2024-01-21T03:19:01.400 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.412+0000 7f697847c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:01.400 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.412+0000 7f697847c640 1 -- 172.21.15.177:0/2163666717 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6379a402-760a-4f07-9491-0918752a18fb", "id": 3} v 0) v1 -- 0x7f697004e970 con 0x7f69700b2310 2024-01-21T03:19:01.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.489+0000 7f6966ffd640 1 -- 172.21.15.177:0/2163666717 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6379a402-760a-4f07-9491-0918752a18fb", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6968017070 con 0x7f69700b2310 2024-01-21T03:19:01.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.489+0000 7f6966ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6379a402-760a-4f07-9491-0918752a18fb", "id": 3}] 2024-01-21T03:19:01.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.489+0000 7f6966ffd640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:01.477 INFO:teuthology.orchestra.run.smithi177.stdout:3 2024-01-21T03:19:01.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 10 librados: watch_flush enter 2024-01-21T03:19:01.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 10 librados: watch_flush exit 2024-01-21T03:19:01.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 20 client.4129.objecter shutdown clearing up homeless session... 2024-01-21T03:19:01.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 10 client.4129.objecter successfully canceled tick 2024-01-21T03:19:01.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 10 monclient: shutdown 2024-01-21T03:19:01.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:01.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 1 -- 172.21.15.177:0/2163666717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69700b2310 msgr2=0x7f6970136c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 1 --2- 172.21.15.177:0/2163666717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69700b2310 0x7f6970136c30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f696800ca70 tx=0x7f696800cf30 comp rx=0 tx=0).stop 2024-01-21T03:19:01.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 1 -- 172.21.15.177:0/2163666717 shutdown_connections 2024-01-21T03:19:01.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 1 --2- 172.21.15.177:0/2163666717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69700b2310 0x7f6970136c30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 1 --2- 172.21.15.177:0/2163666717 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f69700a9d10 0x7f6970137540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 1 --2- 172.21.15.177:0/2163666717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69700a8b20 0x7f6970144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 1 -- 172.21.15.177:0/2163666717 >> 172.21.15.177:0/2163666717 conn(0x7f6970066fb0 msgr2=0x7f69700b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:01.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 1 -- 172.21.15.177:0/2163666717 shutdown_connections 2024-01-21T03:19:01.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 1 -- 172.21.15.177:0/2163666717 wait complete. 2024-01-21T03:19:01.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.490+0000 7f697847c640 1 librados: shutdown 2024-01-21T03:19:01.488 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new dc3602e1-b59e-4489-9882-40c914e2a0d7 4 2024-01-21T03:19:01.494 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:01.507+0000 7f1ba1a43040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-21T03:19:01.545 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:01.559+0000 7f7ae9c13040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-21T03:19:01.589 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:01.603+0000 7f7ae9c13040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-21T03:19:01.593 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.605+0000 7f99832c2640 10 monclient: get_monmap_and_config 2024-01-21T03:19:01.593 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.605+0000 7f99832c2640 10 monclient: build_initial_monmap 2024-01-21T03:19:01.593 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.605+0000 7f99832c2640 10 monclient: monmap: 2024-01-21T03:19:01.593 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:01.593 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:01.593 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:19:01.607726+0000 2024-01-21T03:19:01.593 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:19:01.607726+0000 2024-01-21T03:19:01.594 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:01.594 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.594 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:01.594 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:01.594 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:01.594 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:01.594 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:01.594 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.594 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.606+0000 7f99832c2640 1 Processor -- start 2024-01-21T03:19:01.594 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.606+0000 7f99832c2640 1 -- start start 2024-01-21T03:19:01.594 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.606+0000 7f99832c2640 10 monclient: init 2024-01-21T03:19:01.599 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f99832c2640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:01.599 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f99832c2640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-01-21T03:19:01.599 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f99832c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f997c0a8b20 addr v1:172.21.15.177:6789/0 2024-01-21T03:19:01.599 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f99832c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f997c0a9d10 addr v1:172.21.15.7:6789/0 2024-01-21T03:19:01.599 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f99832c2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f997c0b2310 0x7f997c13f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:01.599 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f99832c2640 10 monclient(hunting): picked mon.noname-c con 0x7f997c0b2310 addr v2:172.21.15.177:3300/0 2024-01-21T03:19:01.599 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f99832c2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997c1456f0 con 0x7f997c0a9d10 2024-01-21T03:19:01.599 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f99832c2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997c0a98c0 con 0x7f997c0a8b20 2024-01-21T03:19:01.600 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f99832c2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:01.600 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f99832c2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f997c146290 con 0x7f997c0b2310 2024-01-21T03:19:01.600 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f99832c2640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:01.600 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f99832c2640 10 monclient(hunting): authenticate will time out at 1292.994312s 2024-01-21T03:19:01.600 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f998185f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f997c0b2310 0x7f997c13f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:01.600 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.607+0000 7f998085d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f997c0a9d10 0x7f997c13f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.177:38198/0 (socket says 172.21.15.177:38198) 2024-01-21T03:19:01.601 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.608+0000 7f998085d640 1 -- 172.21.15.177:0/317987687 learned_addr learned my addr 172.21.15.177:0/317987687 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-01-21T03:19:01.601 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.608+0000 7f998185f640 10 monclient(hunting): get_auth_request con 0x7f997c0b2310 auth_method 0 2024-01-21T03:19:01.601 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.608+0000 7f998185f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:01.601 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.608+0000 7f998185f640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.601 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.608+0000 7f998185f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.601 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.608+0000 7f998185f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:01.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.608+0000 7f998185f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:01.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.608+0000 7f998185f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:01.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.610+0000 7f998185f640 10 monclient(hunting): handle_auth_done global_id 4132 payload 274 2024-01-21T03:19:01.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.610+0000 7f998185f640 1 -- 172.21.15.177:0/317987687 >> v1:172.21.15.177:6789/0 conn(0x7f997c0a8b20 legacy=0x7f997c145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:01.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.610+0000 7f998185f640 1 -- 172.21.15.177:0/317987687 >> v1:172.21.15.7:6789/0 conn(0x7f997c0a9d10 legacy=0x7f997c13f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.610+0000 7f998185f640 10 monclient: _finish_hunting 0 2024-01-21T03:19:01.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.610+0000 7f998185f640 1 monclient: found mon.noname-c 2024-01-21T03:19:01.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.610+0000 7f998185f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:01.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.610+0000 7f998185f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:19:01.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.610+0000 7f998185f640 1 -- 172.21.15.177:0/317987687 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997c1463d0 con 0x7f997c0b2310 2024-01-21T03:19:01.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.612+0000 7f998185f640 1 --2- 172.21.15.177:0/317987687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c0b2310 0x7f997c13f790 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9974009d90 tx=0x7f997402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=237b47d05762d9d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:01.608 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f9973fff640 1 -- 172.21.15.177:0/317987687 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997402cc60 con 0x7f997c0b2310 2024-01-21T03:19:01.608 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f9973fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:01.608 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f9973fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:19:01.608 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f9973fff640 10 monclient: dump: 2024-01-21T03:19:01.608 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:01.608 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:01.608 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.608 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.609 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:01.609 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.609 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:01.609 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:01.609 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:01.609 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.609 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f9973fff640 10 monclient: _finish_auth 0 2024-01-21T03:19:01.609 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f9973fff640 10 monclient: _check_auth_tickets 2024-01-21T03:19:01.609 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f9973fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:01.609 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f9973fff640 1 -- 172.21.15.177:0/317987687 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997402cdc0 con 0x7f997c0b2310 2024-01-21T03:19:01.609 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f99832c2640 5 monclient: authenticate success, global_id 4132 2024-01-21T03:19:01.609 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f99832c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f9973fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f9973fff640 1 -- 172.21.15.177:0/317987687 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974034660 con 0x7f997c0b2310 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f9973fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f9973fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f9973fff640 10 monclient: dump: 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f99832c2640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:01.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.616+0000 7f99832c2640 4 set_mon_vals no callback set 2024-01-21T03:19:01.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.617+0000 7f99832c2640 10 monclient: shutdown 2024-01-21T03:19:01.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.617+0000 7f99832c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:01.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.617+0000 7f99832c2640 1 -- 172.21.15.177:0/317987687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c0b2310 msgr2=0x7f997c13f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.617+0000 7f99832c2640 1 --2- 172.21.15.177:0/317987687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c0b2310 0x7f997c13f790 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9974009d90 tx=0x7f997402beb0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.617+0000 7f99832c2640 1 -- 172.21.15.177:0/317987687 shutdown_connections 2024-01-21T03:19:01.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.617+0000 7f99832c2640 1 --2- 172.21.15.177:0/317987687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c0b2310 0x7f997c13f790 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.617+0000 7f99832c2640 1 -- 172.21.15.177:0/317987687 >> 172.21.15.177:0/317987687 conn(0x7f997c066fb0 msgr2=0x7f997c0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:01.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.617+0000 7f99832c2640 1 -- 172.21.15.177:0/317987687 shutdown_connections 2024-01-21T03:19:01.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.617+0000 7f99832c2640 1 -- 172.21.15.177:0/317987687 wait complete. 2024-01-21T03:19:01.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.618+0000 7f99832c2640 10 monclient: build_initial_monmap 2024-01-21T03:19:01.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.618+0000 7f99832c2640 10 monclient: monmap: 2024-01-21T03:19:01.612 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:01.612 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:01.612 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-01-21T03:19:01.612 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-01-21T03:19:01.612 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:01.612 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.612 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:01.612 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:01.612 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:01.612 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.612 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.618+0000 7f99832c2640 1 librados: starting msgr at 2024-01-21T03:19:01.612 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.618+0000 7f99832c2640 1 librados: starting objecter 2024-01-21T03:19:01.612 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.618+0000 7f99832c2640 1 Processor -- start 2024-01-21T03:19:01.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.618+0000 7f99832c2640 1 -- start start 2024-01-21T03:19:01.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.618+0000 7f99832c2640 1 librados: setting wanted keys 2024-01-21T03:19:01.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.618+0000 7f99832c2640 1 librados: calling monclient init 2024-01-21T03:19:01.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.618+0000 7f99832c2640 10 monclient: init 2024-01-21T03:19:01.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:01.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T03:19:01.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f997c0a8b20 0x7f997c136fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:01.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 10 monclient(hunting): picked mon.noname-c con 0x7f997c0a8b20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:01.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f997c0a9d10 0x7f997c1402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:01.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 10 monclient(hunting): picked mon.noname-a con 0x7f997c0a9d10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:01.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c0b2310 0x7f997c1407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:01.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 10 monclient(hunting): picked mon.noname-b con 0x7f997c0b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:01.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:01.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997c1385c0 con 0x7f997c0a9d10 2024-01-21T03:19:01.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:01.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997c137a00 con 0x7f997c0b2310 2024-01-21T03:19:01.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:01.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997c1387a0 con 0x7f997c0a8b20 2024-01-21T03:19:01.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:01.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f99832c2640 10 monclient(hunting): authenticate will time out at 1293.006283s 2024-01-21T03:19:01.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f998085d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f997c0a9d10 0x7f997c1402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:01.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f998185f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c0b2310 0x7f997c1407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:01.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.619+0000 7f998105e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f997c0a8b20 0x7f997c136fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:01.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998085d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f997c0a9d10 0x7f997c1402a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.177:59318/0 (socket says 172.21.15.177:59318) 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998085d640 1 -- 172.21.15.177:0/238829972 learned_addr learned my addr 172.21.15.177:0/238829972 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998085d640 10 monclient(hunting): get_auth_request con 0x7f997c0a9d10 auth_method 0 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998185f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c0b2310 0x7f997c1407e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:46728/0 (socket says 172.21.15.177:46728) 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998085d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998085d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998085d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998185f640 10 monclient(hunting): get_auth_request con 0x7f997c0b2310 auth_method 0 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998185f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998185f640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998185f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998105e640 10 monclient(hunting): get_auth_request con 0x7f997c0a8b20 auth_method 0 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998105e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998105e640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998105e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998185f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:01.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998185f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:01.616 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998185f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:01.616 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998085d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:01.616 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998085d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:01.616 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.620+0000 7f998085d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:01.616 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.621+0000 7f998105e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:01.616 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.621+0000 7f998105e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:01.616 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.621+0000 7f998105e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:01.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.623+0000 7f998185f640 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2024-01-21T03:19:01.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.623+0000 7f998185f640 1 -- 172.21.15.177:0/238829972 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f997c0a8b20 msgr2=0x7f997c136fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.623+0000 7f998185f640 1 --2- 172.21.15.177:0/238829972 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f997c0a8b20 0x7f997c136fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.623+0000 7f998185f640 1 -- 172.21.15.177:0/238829972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f997c0a9d10 msgr2=0x7f997c1402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.623+0000 7f998185f640 1 --2- 172.21.15.177:0/238829972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f997c0a9d10 0x7f997c1402a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.623+0000 7f998185f640 10 monclient: _finish_hunting 0 2024-01-21T03:19:01.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.623+0000 7f998185f640 1 monclient: found mon.noname-b 2024-01-21T03:19:01.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.623+0000 7f998185f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:01.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.623+0000 7f998185f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:19:01.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.623+0000 7f998185f640 1 -- 172.21.15.177:0/238829972 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9974009a70 con 0x7f997c0b2310 2024-01-21T03:19:01.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.625+0000 7f998185f640 1 --2- 172.21.15.177:0/238829972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c0b2310 0x7f997c1407e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f997c0b89d0 tx=0x7f9974004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:01.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.625+0000 7f9971ffb640 10 client.?.objecter ms_handle_connect 0x7f997c0b2310 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.625+0000 7f9971ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f9971ffb640 1 -- 172.21.15.177:0/238829972 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997403e610 con 0x7f997c0b2310 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f9971ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f9971ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f9971ffb640 10 monclient: dump: 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f9971ffb640 10 monclient: _finish_auth 0 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f9971ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f9971ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f9971ffb640 1 -- 172.21.15.177:0/238829972 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997403e770 con 0x7f997c0b2310 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f99832c2640 5 monclient: authenticate success, global_id 4135 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f9971ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f99832c2640 10 monclient: _renew_subs 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f9971ffb640 1 -- 172.21.15.177:0/238829972 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997403eb20 con 0x7f997c0b2310 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f99832c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f99832c2640 1 -- 172.21.15.177:0/238829972 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997c138a80 con 0x7f997c0b2310 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f9971ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f99737fe640 4 set_mon_vals no callback set 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f9971ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:01.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f9971ffb640 10 monclient: dump: 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f99832c2640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f99832c2640 10 monclient: _renew_subs 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f99832c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:01.622 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f99832c2640 1 -- 172.21.15.177:0/238829972 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997c142e10 con 0x7f997c0b2310 2024-01-21T03:19:01.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.629+0000 7f99832c2640 1 librados: init done 2024-01-21T03:19:01.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.630+0000 7f99832c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:01.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.630+0000 7f99832c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:01.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.630+0000 7f99832c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:01.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.630+0000 7f99832c2640 1 -- 172.21.15.177:0/238829972 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f997c142b30 con 0x7f997c0b2310 2024-01-21T03:19:01.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.633+0000 7f9971ffb640 1 -- 172.21.15.177:0/238829972 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f997403b050 con 0x7f997c0b2310 2024-01-21T03:19:01.624 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.633+0000 7f9971ffb640 1 -- 172.21.15.177:0/238829972 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f9974053cc0 con 0x7f997c0b2310 2024-01-21T03:19:01.624 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.633+0000 7f9971ffb640 10 client.4135.objecter ms_dispatch 0x7f997c000b90 osd_map(6..6 src has 1..6) v4 2024-01-21T03:19:01.624 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.633+0000 7f9971ffb640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-01-21T03:19:01.624 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.633+0000 7f9971ffb640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-01-21T03:19:01.624 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.633+0000 7f9971ffb640 20 client.4135.objecter dump_active .. 0 homeless 2024-01-21T03:19:01.644 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:01.657+0000 7f1ba1a43040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-21T03:19:01.688 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:01.701+0000 7f1ba1a43040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-21T03:19:01.722 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.735+0000 7f9971ffb640 1 -- 172.21.15.177:0/238829972 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f997c142b30 con 0x7f997c0b2310 2024-01-21T03:19:01.722 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.735+0000 7f9971ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:01.722 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.735+0000 7f9971ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:01.738 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:01.751+0000 7f7ae9c13040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-21T03:19:01.795 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.808+0000 7f99832c2640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dc3602e1-b59e-4489-9882-40c914e2a0d7", "id": 4}] 2024-01-21T03:19:01.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.808+0000 7f99832c2640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dc3602e1-b59e-4489-9882-40c914e2a0d7", "id": 4}] 2024-01-21T03:19:01.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.808+0000 7f99832c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:01.796 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.808+0000 7f99832c2640 1 -- 172.21.15.177:0/238829972 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dc3602e1-b59e-4489-9882-40c914e2a0d7", "id": 4} v 0) v1 -- 0x7f997c142fa0 con 0x7f997c0b2310 2024-01-21T03:19:01.837 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:01.850+0000 7f1ba1a43040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-21T03:19:01.865 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.878+0000 7f9971ffb640 1 -- 172.21.15.177:0/238829972 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dc3602e1-b59e-4489-9882-40c914e2a0d7", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f997404e060 con 0x7f997c0b2310 2024-01-21T03:19:01.865 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.878+0000 7f9971ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dc3602e1-b59e-4489-9882-40c914e2a0d7", "id": 4}] 2024-01-21T03:19:01.865 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.878+0000 7f9971ffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:01.865 INFO:teuthology.orchestra.run.smithi177.stdout:4 2024-01-21T03:19:01.866 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.878+0000 7f99832c2640 10 librados: watch_flush enter 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.878+0000 7f99832c2640 10 librados: watch_flush exit 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.878+0000 7f99832c2640 20 client.4135.objecter shutdown clearing up homeless session... 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.878+0000 7f99832c2640 10 client.4135.objecter successfully canceled tick 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.878+0000 7f99832c2640 10 monclient: shutdown 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.879+0000 7f99832c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.879+0000 7f99832c2640 1 -- 172.21.15.177:0/238829972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c0b2310 msgr2=0x7f997c1407e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.879+0000 7f99832c2640 1 --2- 172.21.15.177:0/238829972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c0b2310 0x7f997c1407e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f997c0b89d0 tx=0x7f9974004a40 comp rx=0 tx=0).stop 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.879+0000 7f99832c2640 1 -- 172.21.15.177:0/238829972 shutdown_connections 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.879+0000 7f99832c2640 1 --2- 172.21.15.177:0/238829972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c0b2310 0x7f997c1407e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.879+0000 7f99832c2640 1 --2- 172.21.15.177:0/238829972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f997c0a9d10 0x7f997c1402a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.879+0000 7f99832c2640 1 --2- 172.21.15.177:0/238829972 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f997c0a8b20 0x7f997c136fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.879+0000 7f99832c2640 1 -- 172.21.15.177:0/238829972 >> 172.21.15.177:0/238829972 conn(0x7f997c066fb0 msgr2=0x7f997c0b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.879+0000 7f99832c2640 1 -- 172.21.15.177:0/238829972 shutdown_connections 2024-01-21T03:19:01.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.879+0000 7f99832c2640 1 -- 172.21.15.177:0/238829972 wait complete. 2024-01-21T03:19:01.868 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.879+0000 7f99832c2640 1 librados: shutdown 2024-01-21T03:19:01.876 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new d42afb34-e1ca-4b21-81e9-78d1c8ea454e 5 2024-01-21T03:19:01.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.988+0000 7f9ba4db0640 10 monclient: get_monmap_and_config 2024-01-21T03:19:01.976 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.988+0000 7f9ba4db0640 10 monclient: build_initial_monmap 2024-01-21T03:19:01.976 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.988+0000 7f9ba4db0640 10 monclient: monmap: 2024-01-21T03:19:01.976 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:01.976 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:01.976 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:19:01.990653+0000 2024-01-21T03:19:01.976 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:19:01.990653+0000 2024-01-21T03:19:01.976 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:01.976 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.976 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:01.976 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:01.976 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:01.976 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:01.976 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:01.977 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.978 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.989+0000 7f9ba4db0640 1 Processor -- start 2024-01-21T03:19:01.978 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.989+0000 7f9ba4db0640 1 -- start start 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.989+0000 7f9ba4db0640 10 monclient: init 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9ba4db0640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9ba4db0640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9ba4db0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ba00a8b20 0x7f9ba0145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9ba4db0640 10 monclient(hunting): picked mon.noname-b con 0x7f9ba00a8b20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9ba4db0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ba00a9d10 addr v1:172.21.15.7:6789/0 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9ba4db0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ba00b2310 addr v1:172.21.15.177:6789/0 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9ba4db0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba01456f0 con 0x7f9ba00a9d10 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9ba4db0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba00a98c0 con 0x7f9ba00b2310 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9ba4db0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9ba4db0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba01460c0 con 0x7f9ba00a8b20 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9ba4db0640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9ba4db0640 10 monclient(hunting): authenticate will time out at 1293.377277s 2024-01-21T03:19:01.979 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9b9dd9b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9ba00a9d10 0x7f9ba013f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.177:38210/0 (socket says 172.21.15.177:38210) 2024-01-21T03:19:01.980 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9b9dd9b640 1 -- 172.21.15.177:0/2513033276 learned_addr learned my addr 172.21.15.177:0/2513033276 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-01-21T03:19:01.980 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.990+0000 7f9b9e59c640 1 --2- 172.21.15.177:0/2513033276 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ba00a8b20 0x7f9ba0145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:01.980 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.991+0000 7f9b9e59c640 10 monclient(hunting): get_auth_request con 0x7f9ba00a8b20 auth_method 0 2024-01-21T03:19:01.980 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.991+0000 7f9b9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:01.980 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.991+0000 7f9b9e59c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.980 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.991+0000 7f9b9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.980 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.991+0000 7f9b9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:01.980 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.991+0000 7f9b9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:01.980 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.991+0000 7f9b9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:01.981 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.993+0000 7f9b9e59c640 10 monclient(hunting): handle_auth_done global_id 4127 payload 274 2024-01-21T03:19:01.981 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.993+0000 7f9b9e59c640 1 -- 172.21.15.177:0/2513033276 >> v1:172.21.15.177:6789/0 conn(0x7f9ba00b2310 legacy=0x7f9ba014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.981 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.993+0000 7f9b9e59c640 1 -- 172.21.15.177:0/2513033276 >> v1:172.21.15.7:6789/0 conn(0x7f9ba00a9d10 legacy=0x7f9ba013f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.981 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.993+0000 7f9b9d59a640 1 -- 172.21.15.177:0/2513033276 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159902150 0 0) 0x7f9ba01456f0 con 0x7f9ba00a9d10 2024-01-21T03:19:01.981 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.993+0000 7f9b9e59c640 10 monclient: _finish_hunting 0 2024-01-21T03:19:01.981 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.993+0000 7f9b9e59c640 1 monclient: found mon.noname-b 2024-01-21T03:19:01.981 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.993+0000 7f9b9e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:01.981 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.993+0000 7f9b9e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:19:01.981 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.993+0000 7f9b9e59c640 1 -- 172.21.15.177:0/2513033276 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba0146200 con 0x7f9ba00a8b20 2024-01-21T03:19:01.981 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.993+0000 7f9b9d59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-21T03:19:01.982 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.995+0000 7f9b9e59c640 1 --2- 172.21.15.177:0/2513033276 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ba00a8b20 0x7f9ba0145b80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9b940097d0 tx=0x7f9b9402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=795536f5adc389d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:01.984 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.997+0000 7f9b9d59a640 1 -- 172.21.15.177:0/2513033276 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94031890 con 0x7f9ba00a8b20 2024-01-21T03:19:01.984 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.997+0000 7f9b9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.997+0000 7f9b9d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.997+0000 7f9b9d59a640 10 monclient: dump: 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.997+0000 7f9b9d59a640 10 monclient: _finish_auth 0 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.997+0000 7f9b9d59a640 10 monclient: _check_auth_tickets 2024-01-21T03:19:01.985 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.997+0000 7f9b9d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:01.986 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.997+0000 7f9ba4db0640 5 monclient: authenticate success, global_id 4127 2024-01-21T03:19:01.986 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.997+0000 7f9ba4db0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:19:01.986 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.999+0000 7f9b9d59a640 1 -- 172.21.15.177:0/2513033276 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b940319f0 con 0x7f9ba00a8b20 2024-01-21T03:19:01.986 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.999+0000 7f9b9d59a640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:01.986 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.999+0000 7f9b9d59a640 1 -- 172.21.15.177:0/2513033276 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94031d80 con 0x7f9ba00a8b20 2024-01-21T03:19:01.986 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.999+0000 7f9ba4db0640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:01.986 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.999+0000 7f9ba4db0640 4 set_mon_vals no callback set 2024-01-21T03:19:01.986 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.999+0000 7f9b9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:01.987 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.999+0000 7f9b9d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:19:01.987 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:01.999+0000 7f9b9d59a640 10 monclient: dump: 2024-01-21T03:19:01.987 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:01.987 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:01.987 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.987 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:01.987 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:01.987 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.987 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:01.987 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:01.987 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:01.987 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.991 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.000+0000 7f9ba4db0640 10 monclient: shutdown 2024-01-21T03:19:01.991 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.000+0000 7f9ba4db0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:01.991 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.000+0000 7f9ba4db0640 1 -- 172.21.15.177:0/2513033276 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ba00a8b20 msgr2=0x7f9ba0145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.992 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.000+0000 7f9ba4db0640 1 --2- 172.21.15.177:0/2513033276 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ba00a8b20 0x7f9ba0145b80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9b940097d0 tx=0x7f9b9402b930 comp rx=0 tx=0).stop 2024-01-21T03:19:01.992 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.000+0000 7f9ba4db0640 1 -- 172.21.15.177:0/2513033276 shutdown_connections 2024-01-21T03:19:01.992 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.000+0000 7f9ba4db0640 1 --2- 172.21.15.177:0/2513033276 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ba00a8b20 0x7f9ba0145b80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.992 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.000+0000 7f9ba4db0640 1 -- 172.21.15.177:0/2513033276 >> 172.21.15.177:0/2513033276 conn(0x7f9ba0066fb0 msgr2=0x7f9ba00b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:01.992 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.000+0000 7f9ba4db0640 1 -- 172.21.15.177:0/2513033276 shutdown_connections 2024-01-21T03:19:01.992 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.000+0000 7f9ba4db0640 1 -- 172.21.15.177:0/2513033276 wait complete. 2024-01-21T03:19:01.992 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.001+0000 7f9ba4db0640 10 monclient: build_initial_monmap 2024-01-21T03:19:01.992 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.001+0000 7f9ba4db0640 10 monclient: monmap: 2024-01-21T03:19:01.992 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:01.992 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:01.992 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-01-21T03:19:01.992 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-01-21T03:19:01.992 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:01.993 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:01.993 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:01.993 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:01.993 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:01.993 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:01.993 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.001+0000 7f9ba4db0640 1 librados: starting msgr at 2024-01-21T03:19:01.993 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.001+0000 7f9ba4db0640 1 librados: starting objecter 2024-01-21T03:19:01.993 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.001+0000 7f9ba4db0640 1 Processor -- start 2024-01-21T03:19:01.993 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.001+0000 7f9ba4db0640 1 -- start start 2024-01-21T03:19:01.993 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.001+0000 7f9ba4db0640 1 librados: setting wanted keys 2024-01-21T03:19:01.993 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.001+0000 7f9ba4db0640 1 librados: calling monclient init 2024-01-21T03:19:01.993 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.001+0000 7f9ba4db0640 10 monclient: init 2024-01-21T03:19:01.993 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.001+0000 7f9ba4db0640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.001+0000 7f9ba4db0640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ba00a8b20 0x7f9ba0140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 10 monclient(hunting): picked mon.noname-c con 0x7f9ba00a8b20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba00a9d10 0x7f9ba0143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 10 monclient(hunting): picked mon.noname-b con 0x7f9ba00a9d10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ba00b2310 0x7f9ba0143ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 10 monclient(hunting): picked mon.noname-a con 0x7f9ba00b2310 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba0141bc0 con 0x7f9ba00b2310 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba0142c80 con 0x7f9ba00a9d10 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba0142f60 con 0x7f9ba00a8b20 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:01.994 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9ba4db0640 10 monclient(hunting): authenticate will time out at 1293.389177s 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9b9dd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba00a9d10 0x7f9ba0143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9b9ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ba00b2310 0x7f9ba0143ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9b9e59c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ba00a8b20 0x7f9ba0140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9b9e59c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ba00a8b20 0x7f9ba0140e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.177:39282/0 (socket says 172.21.15.177:39282) 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9b9e59c640 1 -- 172.21.15.177:0/3406266686 learned_addr learned my addr 172.21.15.177:0/3406266686 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9b9e59c640 10 monclient(hunting): get_auth_request con 0x7f9ba00a8b20 auth_method 0 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9b9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9b9e59c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.002+0000 7f9b9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9ed9d640 10 monclient(hunting): get_auth_request con 0x7f9ba00b2310 auth_method 0 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9dd9b640 10 monclient(hunting): get_auth_request con 0x7f9ba00a9d10 auth_method 0 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:01.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9dd9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:01.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:01.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:01.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:01.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:01.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:01.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:01.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:01.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:01.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:01.996 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.003+0000 7f9b9dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:01.997 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.006+0000 7f9b9ed9d640 10 monclient(hunting): handle_auth_done global_id 4140 payload 824 2024-01-21T03:19:01.997 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.006+0000 7f9b9ed9d640 1 -- 172.21.15.177:0/3406266686 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ba00a8b20 msgr2=0x7f9ba0140e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.997 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.006+0000 7f9b9ed9d640 1 --2- 172.21.15.177:0/3406266686 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ba00a8b20 0x7f9ba0140e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.997 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.006+0000 7f9b9ed9d640 1 -- 172.21.15.177:0/3406266686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba00a9d10 msgr2=0x7f9ba0143660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:01.997 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.006+0000 7f9b9ed9d640 1 --2- 172.21.15.177:0/3406266686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba00a9d10 0x7f9ba0143660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:01.997 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.006+0000 7f9b9ed9d640 10 monclient: _finish_hunting 0 2024-01-21T03:19:01.997 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.006+0000 7f9b9ed9d640 1 monclient: found mon.noname-a 2024-01-21T03:19:01.998 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.006+0000 7f9b9ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:01.998 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.006+0000 7f9b9ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:19:01.998 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.006+0000 7f9b9ed9d640 1 -- 172.21.15.177:0/3406266686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b94009400 con 0x7f9ba00b2310 2024-01-21T03:19:01.998 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.009+0000 7f9b9ed9d640 1 --2- 172.21.15.177:0/3406266686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ba00b2310 0x7f9ba0143ba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00b41d0 tx=0x7f9b9800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:01.998 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.009+0000 7f9b8effd640 10 client.?.objecter ms_handle_connect 0x7f9ba00b2310 2024-01-21T03:19:01.998 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.009+0000 7f9b8effd640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:02.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b8effd640 1 -- 172.21.15.177:0/3406266686 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b98003b70 con 0x7f9ba00b2310 2024-01-21T03:19:02.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:02.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:19:02.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b8effd640 10 monclient: dump: 2024-01-21T03:19:02.002 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:02.002 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:02.002 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.002 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.002 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:02.002 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b8effd640 10 monclient: _finish_auth 0 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b8effd640 10 monclient: _check_auth_tickets 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b8effd640 1 -- 172.21.15.177:0/3406266686 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b98003cd0 con 0x7f9ba00b2310 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b8effd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b8effd640 1 -- 172.21.15.177:0/3406266686 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b98005310 con 0x7f9ba00b2310 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b8effd640 10 monclient: dump: 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:02.003 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9ba4db0640 5 monclient: authenticate success, global_id 4140 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9ba4db0640 10 monclient: _renew_subs 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9ba4db0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9b9cd99640 4 set_mon_vals no callback set 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9ba4db0640 1 -- 172.21.15.177:0/3406266686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba0143240 con 0x7f9ba00b2310 2024-01-21T03:19:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9ba4db0640 10 client.4140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:02.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9ba4db0640 10 monclient: _renew_subs 2024-01-21T03:19:02.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9ba4db0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:02.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9ba4db0640 1 -- 172.21.15.177:0/3406266686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba01441a0 con 0x7f9ba00b2310 2024-01-21T03:19:02.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.013+0000 7f9ba4db0640 1 librados: init done 2024-01-21T03:19:02.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.014+0000 7f9ba4db0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:02.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.014+0000 7f9ba4db0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:02.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.014+0000 7f9ba4db0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:02.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.014+0000 7f9ba4db0640 1 -- 172.21.15.177:0/3406266686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba0141d70 con 0x7f9ba00b2310 2024-01-21T03:19:02.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.017+0000 7f9b8effd640 1 -- 172.21.15.177:0/3406266686 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f9b9800a640 con 0x7f9ba00b2310 2024-01-21T03:19:02.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.136+0000 7f9b8effd640 1 -- 172.21.15.177:0/3406266686 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f9b98003e40 con 0x7f9ba00b2310 2024-01-21T03:19:02.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.136+0000 7f9b8effd640 10 client.4140.objecter ms_dispatch 0x7f9ba0000b90 osd_map(7..7 src has 1..7) v4 2024-01-21T03:19:02.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.136+0000 7f9b8effd640 3 client.4140.objecter handle_osd_map got epochs [7,7] > 0 2024-01-21T03:19:02.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.136+0000 7f9b8effd640 3 client.4140.objecter handle_osd_map decoding full epoch 7 2024-01-21T03:19:02.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.136+0000 7f9b8effd640 20 client.4140.objecter dump_active .. 0 homeless 2024-01-21T03:19:02.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.136+0000 7f9b8effd640 1 -- 172.21.15.177:0/3406266686 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f9ba0141d70 con 0x7f9ba00b2310 2024-01-21T03:19:02.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.136+0000 7f9b8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:02.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.136+0000 7f9b8effd640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:02.196 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.209+0000 7f9ba4db0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d42afb34-e1ca-4b21-81e9-78d1c8ea454e", "id": 5}] 2024-01-21T03:19:02.196 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.209+0000 7f9ba4db0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d42afb34-e1ca-4b21-81e9-78d1c8ea454e", "id": 5}] 2024-01-21T03:19:02.196 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.209+0000 7f9ba4db0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:02.196 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.209+0000 7f9ba4db0640 1 -- 172.21.15.177:0/3406266686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d42afb34-e1ca-4b21-81e9-78d1c8ea454e", "id": 5} v 0) v1 -- 0x7f9ba004e970 con 0x7f9ba00b2310 2024-01-21T03:19:02.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.239+0000 7f9b8effd640 1 -- 172.21.15.177:0/3406266686 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d42afb34-e1ca-4b21-81e9-78d1c8ea454e", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9b9800a9c0 con 0x7f9ba00b2310 2024-01-21T03:19:02.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.239+0000 7f9b8effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d42afb34-e1ca-4b21-81e9-78d1c8ea454e", "id": 5}] 2024-01-21T03:19:02.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.239+0000 7f9b8effd640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:02.227 INFO:teuthology.orchestra.run.smithi177.stdout:5 2024-01-21T03:19:02.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 10 librados: watch_flush enter 2024-01-21T03:19:02.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 10 librados: watch_flush exit 2024-01-21T03:19:02.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 20 client.4140.objecter shutdown clearing up homeless session... 2024-01-21T03:19:02.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 10 client.4140.objecter successfully canceled tick 2024-01-21T03:19:02.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 10 monclient: shutdown 2024-01-21T03:19:02.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:02.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 1 -- 172.21.15.177:0/3406266686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ba00b2310 msgr2=0x7f9ba0143ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:02.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 1 --2- 172.21.15.177:0/3406266686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ba00b2310 0x7f9ba0143ba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00b41d0 tx=0x7f9b9800cb90 comp rx=0 tx=0).stop 2024-01-21T03:19:02.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 1 -- 172.21.15.177:0/3406266686 shutdown_connections 2024-01-21T03:19:02.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 1 --2- 172.21.15.177:0/3406266686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ba00b2310 0x7f9ba0143ba0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 1 --2- 172.21.15.177:0/3406266686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba00a9d10 0x7f9ba0143660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 1 --2- 172.21.15.177:0/3406266686 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ba00a8b20 0x7f9ba0140e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 1 -- 172.21.15.177:0/3406266686 >> 172.21.15.177:0/3406266686 conn(0x7f9ba0066fb0 msgr2=0x7f9ba00b5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:02.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 1 -- 172.21.15.177:0/3406266686 shutdown_connections 2024-01-21T03:19:02.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.240+0000 7f9ba4db0640 1 -- 172.21.15.177:0/3406266686 wait complete. 2024-01-21T03:19:02.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.241+0000 7f9ba4db0640 1 librados: shutdown 2024-01-21T03:19:02.239 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 1aef2850-8b41-4d39-859e-ed9c4df434b0 6 2024-01-21T03:19:02.339 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.352+0000 7f5e3ed89640 10 monclient: get_monmap_and_config 2024-01-21T03:19:02.339 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.352+0000 7f5e3ed89640 10 monclient: build_initial_monmap 2024-01-21T03:19:02.339 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.352+0000 7f5e3ed89640 10 monclient: monmap: 2024-01-21T03:19:02.339 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:02.339 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:02.339 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:19:02.353957+0000 2024-01-21T03:19:02.339 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:19:02.353957+0000 2024-01-21T03:19:02.340 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:02.340 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.340 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:02.340 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:02.340 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:02.340 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:02.340 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:02.340 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.343 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.352+0000 7f5e3ed89640 1 Processor -- start 2024-01-21T03:19:02.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.352+0000 7f5e3ed89640 1 -- start start 2024-01-21T03:19:02.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.352+0000 7f5e3ed89640 10 monclient: init 2024-01-21T03:19:02.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:02.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-01-21T03:19:02.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5e380a8b20 0x7f5e38145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:02.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 10 monclient(hunting): picked mon.noname-a con 0x7f5e380a8b20 addr v2:172.21.15.7:3300/0 2024-01-21T03:19:02.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e380a9d10 0x7f5e3813f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:02.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 10 monclient(hunting): picked mon.noname-b con 0x7f5e380a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:02.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5e380b2310 0x7f5e3813f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:02.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 10 monclient(hunting): picked mon.noname-c con 0x7f5e380b2310 addr v2:172.21.15.177:3300/0 2024-01-21T03:19:02.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:02.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e381456f0 con 0x7f5e380a8b20 2024-01-21T03:19:02.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e380a98c0 con 0x7f5e380b2310 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3813fb00 con 0x7f5e380a9d10 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3ed89640 10 monclient(hunting): authenticate will time out at 1293.740529s 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.353+0000 7f5e3d326640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5e380b2310 0x7f5e3813f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e37fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e380a9d10 0x7f5e3813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3cb25640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5e380a8b20 0x7f5e38145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3d326640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5e380b2310 0x7f5e3813f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:46744/0 (socket says 172.21.15.177:46744) 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3d326640 1 -- 172.21.15.177:0/790232041 learned_addr learned my addr 172.21.15.177:0/790232041 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3d326640 10 monclient(hunting): get_auth_request con 0x7f5e380b2310 auth_method 0 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3d326640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3d326640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:02.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3d326640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3cb25640 10 monclient(hunting): get_auth_request con 0x7f5e380a8b20 auth_method 0 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3cb25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3cb25640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3cb25640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e37fff640 10 monclient(hunting): get_auth_request con 0x7f5e380a9d10 auth_method 0 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e37fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e37fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3d326640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3d326640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.354+0000 7f5e3d326640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.355+0000 7f5e3cb25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.355+0000 7f5e3cb25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.355+0000 7f5e3cb25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.355+0000 7f5e37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:02.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.355+0000 7f5e37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:02.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.355+0000 7f5e37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:02.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.356+0000 7f5e3d326640 10 monclient(hunting): handle_auth_done global_id 4144 payload 274 2024-01-21T03:19:02.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.356+0000 7f5e3d326640 1 -- 172.21.15.177:0/790232041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e380a9d10 msgr2=0x7f5e3813f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:02.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.356+0000 7f5e3d326640 1 --2- 172.21.15.177:0/790232041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e380a9d10 0x7f5e3813f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.356+0000 7f5e3d326640 1 -- 172.21.15.177:0/790232041 >> v2:172.21.15.7:3300/0 conn(0x7f5e380a8b20 msgr2=0x7f5e38145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:02.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.356+0000 7f5e3d326640 1 --2- 172.21.15.177:0/790232041 >> v2:172.21.15.7:3300/0 conn(0x7f5e380a8b20 0x7f5e38145b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.356+0000 7f5e3d326640 10 monclient: _finish_hunting 0 2024-01-21T03:19:02.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.356+0000 7f5e3d326640 1 monclient: found mon.noname-c 2024-01-21T03:19:02.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.356+0000 7f5e3d326640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:02.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.356+0000 7f5e3d326640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:19:02.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.356+0000 7f5e3d326640 1 -- 172.21.15.177:0/790232041 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e381460c0 con 0x7f5e380b2310 2024-01-21T03:19:02.348 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.358+0000 7f5e3d326640 1 --2- 172.21.15.177:0/790232041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e380b2310 0x7f5e3813f5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5e38145660 tx=0x7f5e2802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82cee951b13677c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:02.348 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e377fe640 1 -- 172.21.15.177:0/790232041 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28031870 con 0x7f5e380b2310 2024-01-21T03:19:02.348 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e377fe640 10 monclient: dump: 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e377fe640 10 monclient: _finish_auth 0 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e377fe640 10 monclient: _check_auth_tickets 2024-01-21T03:19:02.349 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e377fe640 1 -- 172.21.15.177:0/790232041 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e280319d0 con 0x7f5e380b2310 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e377fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e377fe640 1 -- 172.21.15.177:0/790232041 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28031d60 con 0x7f5e380b2310 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e3ed89640 5 monclient: authenticate success, global_id 4144 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e3ed89640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e3ed89640 4 set_mon_vals no callback set 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.361+0000 7f5e377fe640 10 monclient: dump: 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:02.350 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.351 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:02.351 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:02.351 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:02.351 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.362+0000 7f5e3ed89640 10 monclient: shutdown 2024-01-21T03:19:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.362+0000 7f5e3ed89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.362+0000 7f5e3ed89640 1 -- 172.21.15.177:0/790232041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e380b2310 msgr2=0x7f5e3813f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.362+0000 7f5e3ed89640 1 --2- 172.21.15.177:0/790232041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e380b2310 0x7f5e3813f5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5e38145660 tx=0x7f5e2802b910 comp rx=0 tx=0).stop 2024-01-21T03:19:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.362+0000 7f5e3ed89640 1 -- 172.21.15.177:0/790232041 shutdown_connections 2024-01-21T03:19:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.362+0000 7f5e3ed89640 1 --2- 172.21.15.177:0/790232041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e380b2310 0x7f5e3813f5c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.362+0000 7f5e3ed89640 1 --2- 172.21.15.177:0/790232041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e380a9d10 0x7f5e3813f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.362+0000 7f5e3ed89640 1 --2- 172.21.15.177:0/790232041 >> v2:172.21.15.7:3300/0 conn(0x7f5e380a8b20 0x7f5e38145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.362+0000 7f5e3ed89640 1 -- 172.21.15.177:0/790232041 >> 172.21.15.177:0/790232041 conn(0x7f5e38066fb0 msgr2=0x7f5e380b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.362+0000 7f5e3ed89640 1 -- 172.21.15.177:0/790232041 shutdown_connections 2024-01-21T03:19:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.362+0000 7f5e3ed89640 1 -- 172.21.15.177:0/790232041 wait complete. 2024-01-21T03:19:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.362+0000 7f5e3ed89640 10 monclient: build_initial_monmap 2024-01-21T03:19:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.362+0000 7f5e3ed89640 10 monclient: monmap: 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.363+0000 7f5e3ed89640 1 librados: starting msgr at 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.363+0000 7f5e3ed89640 1 librados: starting objecter 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.363+0000 7f5e3ed89640 1 Processor -- start 2024-01-21T03:19:02.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.363+0000 7f5e3ed89640 1 -- start start 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.363+0000 7f5e3ed89640 1 librados: setting wanted keys 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.363+0000 7f5e3ed89640 1 librados: calling monclient init 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.363+0000 7f5e3ed89640 10 monclient: init 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e380a8b20 0x7f5e380b62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 10 monclient(hunting): picked mon.noname-a con 0x7f5e380a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e380a9d10 0x7f5e381da8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 10 monclient(hunting): picked mon.noname-c con 0x7f5e380a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e380b2310 0x7f5e38144db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 10 monclient(hunting): picked mon.noname-b con 0x7f5e380b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e381448b0 con 0x7f5e380a8b20 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:02.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3813a8e0 con 0x7f5e380b2310 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e381452f0 con 0x7f5e380a9d10 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3ed89640 10 monclient(hunting): authenticate will time out at 1293.751185s 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3d326640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e380b2310 0x7f5e38144db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3cb25640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e380a8b20 0x7f5e380b62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e37fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e380a9d10 0x7f5e381da8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3d326640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e380b2310 0x7f5e38144db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:46756/0 (socket says 172.21.15.177:46756) 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3d326640 1 -- 172.21.15.177:0/891231012 learned_addr learned my addr 172.21.15.177:0/891231012 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3d326640 10 monclient(hunting): get_auth_request con 0x7f5e380b2310 auth_method 0 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3d326640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3d326640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.364+0000 7f5e3d326640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e3cb25640 10 monclient(hunting): get_auth_request con 0x7f5e380a8b20 auth_method 0 2024-01-21T03:19:02.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e3cb25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e3cb25640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e3cb25640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e37fff640 10 monclient(hunting): get_auth_request con 0x7f5e380a9d10 auth_method 0 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e37fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e37fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e3d326640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e3d326640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e3d326640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e3cb25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e3cb25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e3cb25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:02.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:02.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.365+0000 7f5e37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:02.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.367+0000 7f5e3d326640 10 monclient(hunting): handle_auth_done global_id 4147 payload 824 2024-01-21T03:19:02.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.368+0000 7f5e3d326640 1 -- 172.21.15.177:0/891231012 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e380a9d10 msgr2=0x7f5e381da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:02.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.368+0000 7f5e3d326640 1 --2- 172.21.15.177:0/891231012 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e380a9d10 0x7f5e381da8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.368+0000 7f5e3d326640 1 -- 172.21.15.177:0/891231012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e380a8b20 msgr2=0x7f5e380b62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:02.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.368+0000 7f5e3d326640 1 --2- 172.21.15.177:0/891231012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e380a8b20 0x7f5e380b62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.368+0000 7f5e3d326640 10 monclient: _finish_hunting 0 2024-01-21T03:19:02.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.368+0000 7f5e3d326640 1 monclient: found mon.noname-b 2024-01-21T03:19:02.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.368+0000 7f5e3d326640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:02.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.368+0000 7f5e3d326640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:19:02.360 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.368+0000 7f5e3d326640 1 -- 172.21.15.177:0/891231012 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e280094d0 con 0x7f5e380b2310 2024-01-21T03:19:02.362 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.370+0000 7f5e3d326640 1 --2- 172.21.15.177:0/891231012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e380b2310 0x7f5e38144db0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5e280090b0 tx=0x7f5e280046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:02.362 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.370+0000 7f5e357fa640 10 client.?.objecter ms_handle_connect 0x7f5e380b2310 2024-01-21T03:19:02.362 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.370+0000 7f5e357fa640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:02.362 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e357fa640 1 -- 172.21.15.177:0/891231012 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2803ab60 con 0x7f5e380b2310 2024-01-21T03:19:02.362 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:02.362 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e357fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:02.362 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e357fa640 10 monclient: dump: 2024-01-21T03:19:02.362 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e357fa640 10 monclient: _finish_auth 0 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e357fa640 10 monclient: _check_auth_tickets 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e357fa640 1 -- 172.21.15.177:0/891231012 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2803ace0 con 0x7f5e380b2310 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e3ed89640 5 monclient: authenticate success, global_id 4147 2024-01-21T03:19:02.363 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e3ed89640 10 monclient: _renew_subs 2024-01-21T03:19:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e3ed89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e3ed89640 1 -- 172.21.15.177:0/891231012 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e381379a0 con 0x7f5e380b2310 2024-01-21T03:19:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e357fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e357fa640 1 -- 172.21.15.177:0/891231012 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28044630 con 0x7f5e380b2310 2024-01-21T03:19:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e357fa640 10 monclient: dump: 2024-01-21T03:19:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e36ffd640 4 set_mon_vals no callback set 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e3ed89640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e3ed89640 10 monclient: _renew_subs 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e3ed89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e3ed89640 1 -- 172.21.15.177:0/891231012 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e381444e0 con 0x7f5e380b2310 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.373+0000 7f5e3ed89640 1 librados: init done 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.375+0000 7f5e3ed89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.375+0000 7f5e3ed89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.375+0000 7f5e3ed89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:02.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.375+0000 7f5e3ed89640 1 -- 172.21.15.177:0/891231012 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e38138b10 con 0x7f5e380b2310 2024-01-21T03:19:02.366 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.376+0000 7f5e357fa640 1 -- 172.21.15.177:0/891231012 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f5e28044790 con 0x7f5e380b2310 2024-01-21T03:19:02.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.474+0000 7f5e357fa640 1 -- 172.21.15.177:0/891231012 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f5e2802d040 con 0x7f5e380b2310 2024-01-21T03:19:02.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.474+0000 7f5e357fa640 10 client.4147.objecter ms_dispatch 0x7f5e38000b90 osd_map(8..8 src has 1..8) v4 2024-01-21T03:19:02.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.474+0000 7f5e357fa640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-01-21T03:19:02.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.474+0000 7f5e357fa640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-01-21T03:19:02.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.474+0000 7f5e357fa640 20 client.4147.objecter dump_active .. 0 homeless 2024-01-21T03:19:02.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.474+0000 7f5e357fa640 1 -- 172.21.15.177:0/891231012 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f5e28035020 con 0x7f5e380b2310 2024-01-21T03:19:02.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.474+0000 7f5e357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:02.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.474+0000 7f5e357fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:02.533 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.546+0000 7f5e3ed89640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1aef2850-8b41-4d39-859e-ed9c4df434b0", "id": 6}] 2024-01-21T03:19:02.533 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.546+0000 7f5e3ed89640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1aef2850-8b41-4d39-859e-ed9c4df434b0", "id": 6}] 2024-01-21T03:19:02.534 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.546+0000 7f5e3ed89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:02.534 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.546+0000 7f5e3ed89640 1 -- 172.21.15.177:0/891231012 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1aef2850-8b41-4d39-859e-ed9c4df434b0", "id": 6} v 0) v1 -- 0x7f5e381454b0 con 0x7f5e380b2310 2024-01-21T03:19:02.577 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:02.591+0000 7f7ae9c13040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-21T03:19:02.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.625+0000 7f5e357fa640 1 -- 172.21.15.177:0/891231012 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1aef2850-8b41-4d39-859e-ed9c4df434b0", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5e2804d5d0 con 0x7f5e380b2310 2024-01-21T03:19:02.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.625+0000 7f5e357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1aef2850-8b41-4d39-859e-ed9c4df434b0", "id": 6}] 2024-01-21T03:19:02.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.625+0000 7f5e357fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:02.613 INFO:teuthology.orchestra.run.smithi177.stdout:6 2024-01-21T03:19:02.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.626+0000 7f5e3ed89640 10 librados: watch_flush enter 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.626+0000 7f5e3ed89640 10 librados: watch_flush exit 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.626+0000 7f5e3ed89640 20 client.4147.objecter shutdown clearing up homeless session... 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.626+0000 7f5e3ed89640 10 client.4147.objecter successfully canceled tick 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.626+0000 7f5e3ed89640 10 monclient: shutdown 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.626+0000 7f5e3ed89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.626+0000 7f5e3ed89640 1 -- 172.21.15.177:0/891231012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e380b2310 msgr2=0x7f5e38144db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.626+0000 7f5e3ed89640 1 --2- 172.21.15.177:0/891231012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e380b2310 0x7f5e38144db0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5e280090b0 tx=0x7f5e280046c0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.627+0000 7f5e3ed89640 1 -- 172.21.15.177:0/891231012 shutdown_connections 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.627+0000 7f5e3ed89640 1 --2- 172.21.15.177:0/891231012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e380b2310 0x7f5e38144db0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.627+0000 7f5e3ed89640 1 --2- 172.21.15.177:0/891231012 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e380a9d10 0x7f5e381da8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.627+0000 7f5e3ed89640 1 --2- 172.21.15.177:0/891231012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e380a8b20 0x7f5e380b62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.627+0000 7f5e3ed89640 1 -- 172.21.15.177:0/891231012 >> 172.21.15.177:0/891231012 conn(0x7f5e38066fb0 msgr2=0x7f5e3813b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.627+0000 7f5e3ed89640 1 -- 172.21.15.177:0/891231012 shutdown_connections 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.627+0000 7f5e3ed89640 1 -- 172.21.15.177:0/891231012 wait complete. 2024-01-21T03:19:02.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.627+0000 7f5e3ed89640 1 librados: shutdown 2024-01-21T03:19:02.624 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e 7 2024-01-21T03:19:02.699 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:02.712+0000 7f1ba1a43040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.737+0000 7f678e52c640 10 monclient: get_monmap_and_config 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.737+0000 7f678e52c640 10 monclient: build_initial_monmap 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.737+0000 7f678e52c640 10 monclient: monmap: 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:19:02.739793+0000 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:19:02.739793+0000 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:02.725 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:02.726 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.738+0000 7f678e52c640 1 Processor -- start 2024-01-21T03:19:02.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.738+0000 7f678e52c640 1 -- start start 2024-01-21T03:19:02.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.738+0000 7f678e52c640 10 monclient: init 2024-01-21T03:19:02.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.739+0000 7f678e52c640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:02.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.739+0000 7f678e52c640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-01-21T03:19:02.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.739+0000 7f678e52c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f67880a8b20 0x7f6788145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:02.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.739+0000 7f678e52c640 10 monclient(hunting): picked mon.noname-c con 0x7f67880a8b20 addr v2:172.21.15.177:3300/0 2024-01-21T03:19:02.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.739+0000 7f678e52c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f67880a9d10 addr v1:172.21.15.7:6789/0 2024-01-21T03:19:02.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.739+0000 7f678e52c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67880b2310 addr v1:172.21.15.177:6789/0 2024-01-21T03:19:02.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.739+0000 7f678e52c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67881456f0 con 0x7f67880a9d10 2024-01-21T03:19:02.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.739+0000 7f678e52c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67880a98c0 con 0x7f67880b2310 2024-01-21T03:19:02.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.739+0000 7f678e52c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:02.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.739+0000 7f678e52c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67881460c0 con 0x7f67880a8b20 2024-01-21T03:19:02.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.739+0000 7f678e52c640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:02.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.739+0000 7f678e52c640 10 monclient(hunting): authenticate will time out at 1294.126441s 2024-01-21T03:19:02.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.740+0000 7f6787fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f67880a8b20 0x7f6788145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:02.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.740+0000 7f67877fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f67880a9d10 0x7f678813f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.177:38224/0 (socket says 172.21.15.177:38224) 2024-01-21T03:19:02.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.740+0000 7f67877fe640 1 -- 172.21.15.177:0/1738734856 learned_addr learned my addr 172.21.15.177:0/1738734856 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-01-21T03:19:02.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.740+0000 7f6787fff640 10 monclient(hunting): get_auth_request con 0x7f67880a8b20 auth_method 0 2024-01-21T03:19:02.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.740+0000 7f6787fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:02.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.740+0000 7f6787fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:02.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.740+0000 7f6787fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:02.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.740+0000 7f6787fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:02.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.740+0000 7f6787fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:02.730 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.740+0000 7f6787fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:02.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6786ffd640 1 -- 172.21.15.177:0/1738734856 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866458420 0 0) 0x7f67881456f0 con 0x7f67880a9d10 2024-01-21T03:19:02.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6786ffd640 10 monclient(hunting): _negotiate 2024-01-21T03:19:02.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6786ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:02.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6786ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:02.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6786ffd640 10 monclient(hunting): my global_id is 4149 2024-01-21T03:19:02.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6786ffd640 1 -- 172.21.15.177:0/1738734856 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6770003520 con 0x7f67880a9d10 2024-01-21T03:19:02.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6787fff640 10 monclient(hunting): handle_auth_done global_id 4150 payload 274 2024-01-21T03:19:02.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6787fff640 1 -- 172.21.15.177:0/1738734856 >> v1:172.21.15.177:6789/0 conn(0x7f67880b2310 legacy=0x7f678814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:02.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6787fff640 1 -- 172.21.15.177:0/1738734856 >> v1:172.21.15.7:6789/0 conn(0x7f67880a9d10 legacy=0x7f678813f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:02.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6787fff640 10 monclient: _finish_hunting 0 2024-01-21T03:19:02.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6787fff640 1 monclient: found mon.noname-c 2024-01-21T03:19:02.732 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6787fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:02.732 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6787fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:19:02.732 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.742+0000 7f6787fff640 1 -- 172.21.15.177:0/1738734856 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6788146200 con 0x7f67880a8b20 2024-01-21T03:19:02.733 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.746+0000 7f6787fff640 1 --2- 172.21.15.177:0/1738734856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67880a8b20 0x7f6788145b80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f677c0097a0 tx=0x7f677c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=553b6e260c8ca71 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:02.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f6786ffd640 1 -- 172.21.15.177:0/1738734856 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c02e070 con 0x7f67880a8b20 2024-01-21T03:19:02.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f6786ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:02.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f6786ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:19:02.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f6786ffd640 10 monclient: dump: 2024-01-21T03:19:02.738 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:02.738 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:02.738 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.738 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.738 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f6786ffd640 10 monclient: _finish_auth 0 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f6786ffd640 10 monclient: _check_auth_tickets 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f6786ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f6786ffd640 1 -- 172.21.15.177:0/1738734856 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677c0315a0 con 0x7f67880a8b20 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f6786ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f6786ffd640 1 -- 172.21.15.177:0/1738734856 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c031930 con 0x7f67880a8b20 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f678e52c640 5 monclient: authenticate success, global_id 4150 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f678e52c640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f678e52c640 4 set_mon_vals no callback set 2024-01-21T03:19:02.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f6786ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:02.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f6786ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:02.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f6786ffd640 10 monclient: dump: 2024-01-21T03:19:02.740 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:02.740 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:02.740 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.740 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.740 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:02.740 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.740 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:02.740 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:02.740 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:02.740 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f678e52c640 10 monclient: shutdown 2024-01-21T03:19:02.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f678e52c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:02.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f678e52c640 1 -- 172.21.15.177:0/1738734856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67880a8b20 msgr2=0x7f6788145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:02.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f678e52c640 1 --2- 172.21.15.177:0/1738734856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67880a8b20 0x7f6788145b80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f677c0097a0 tx=0x7f677c02b840 comp rx=0 tx=0).stop 2024-01-21T03:19:02.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f678e52c640 1 -- 172.21.15.177:0/1738734856 shutdown_connections 2024-01-21T03:19:02.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f678e52c640 1 --2- 172.21.15.177:0/1738734856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67880a8b20 0x7f6788145b80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f678e52c640 1 -- 172.21.15.177:0/1738734856 >> 172.21.15.177:0/1738734856 conn(0x7f6788066fb0 msgr2=0x7f67880b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:02.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f678e52c640 1 -- 172.21.15.177:0/1738734856 shutdown_connections 2024-01-21T03:19:02.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.751+0000 7f678e52c640 1 -- 172.21.15.177:0/1738734856 wait complete. 2024-01-21T03:19:02.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.752+0000 7f678e52c640 10 monclient: build_initial_monmap 2024-01-21T03:19:02.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.752+0000 7f678e52c640 10 monclient: monmap: 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.753+0000 7f678e52c640 1 librados: starting msgr at 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.753+0000 7f678e52c640 1 librados: starting objecter 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.753+0000 7f678e52c640 1 Processor -- start 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.753+0000 7f678e52c640 1 -- start start 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.753+0000 7f678e52c640 1 librados: setting wanted keys 2024-01-21T03:19:02.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.753+0000 7f678e52c640 1 librados: calling monclient init 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.753+0000 7f678e52c640 10 monclient: init 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67880a8b20 0x7f67880abc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 10 monclient(hunting): picked mon.noname-a con 0x7f67880a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f67880a9d10 0x7f67880ac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 10 monclient(hunting): picked mon.noname-c con 0x7f67880a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67880b2310 0x7f67881d75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 10 monclient(hunting): picked mon.noname-b con 0x7f67880b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67880ad790 con 0x7f67880a8b20 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67881d4cb0 con 0x7f67880b2310 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:02.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67881d7110 con 0x7f67880a9d10 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678e52c640 10 monclient(hunting): authenticate will time out at 1294.141209s 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f6787fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67880a8b20 0x7f67880abc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f678cac9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67880b2310 0x7f67881d75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f67877fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f67880a9d10 0x7f67880ac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.754+0000 7f6787fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67880a8b20 0x7f67880abc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.177:59358/0 (socket says 172.21.15.177:59358) 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f6787fff640 1 -- 172.21.15.177:0/1697322656 learned_addr learned my addr 172.21.15.177:0/1697322656 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f6787fff640 10 monclient(hunting): get_auth_request con 0x7f67880a8b20 auth_method 0 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f6787fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f6787fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f6787fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f678cac9640 10 monclient(hunting): get_auth_request con 0x7f67880b2310 auth_method 0 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f678cac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f678cac9640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f678cac9640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:02.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f67877fe640 10 monclient(hunting): get_auth_request con 0x7f67880a9d10 auth_method 0 2024-01-21T03:19:02.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f67877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:02.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f67877fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:02.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f67877fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:02.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f6787fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:02.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f6787fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:02.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f6787fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:02.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f678cac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:02.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f678cac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:02.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f678cac9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:02.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f67877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:02.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f67877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:02.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.755+0000 7f67877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:02.749 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.758+0000 7f6787fff640 10 monclient(hunting): handle_auth_done global_id 4152 payload 824 2024-01-21T03:19:02.749 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.758+0000 7f6787fff640 1 -- 172.21.15.177:0/1697322656 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f67880a9d10 msgr2=0x7f67880ac8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:02.749 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.758+0000 7f6787fff640 1 --2- 172.21.15.177:0/1697322656 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f67880a9d10 0x7f67880ac8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.758+0000 7f6787fff640 1 -- 172.21.15.177:0/1697322656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67880b2310 msgr2=0x7f67881d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:02.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.758+0000 7f6787fff640 1 --2- 172.21.15.177:0/1697322656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67880b2310 0x7f67881d75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.758+0000 7f6787fff640 10 monclient: _finish_hunting 0 2024-01-21T03:19:02.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.758+0000 7f6787fff640 1 monclient: found mon.noname-a 2024-01-21T03:19:02.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.758+0000 7f6787fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:02.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.758+0000 7f6787fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:19:02.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.758+0000 7f6787fff640 1 -- 172.21.15.177:0/1697322656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677c009400 con 0x7f67880a8b20 2024-01-21T03:19:02.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.760+0000 7f6787fff640 1 --2- 172.21.15.177:0/1697322656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67880a8b20 0x7f67880abc60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f677c0097a0 tx=0x7f677c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:02.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.760+0000 7f6784ff9640 10 client.?.objecter ms_handle_connect 0x7f67880a8b20 2024-01-21T03:19:02.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.760+0000 7f6784ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:02.751 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f6784ff9640 1 -- 172.21.15.177:0/1697322656 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c032840 con 0x7f67880a8b20 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f6784ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f6784ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f6784ff9640 10 monclient: dump: 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f6784ff9640 10 monclient: _finish_auth 0 2024-01-21T03:19:02.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f6784ff9640 10 monclient: _check_auth_tickets 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f6784ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f6784ff9640 1 -- 172.21.15.177:0/1697322656 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677c0329a0 con 0x7f67880a8b20 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f678e52c640 5 monclient: authenticate success, global_id 4152 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f678e52c640 10 monclient: _renew_subs 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f678e52c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f678e52c640 1 -- 172.21.15.177:0/1697322656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67880ad9e0 con 0x7f67880a8b20 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f6784ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f6784ff9640 1 -- 172.21.15.177:0/1697322656 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c032d50 con 0x7f67880a8b20 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f6784ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f6784ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f6784ff9640 10 monclient: dump: 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.753 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f67867fc640 4 set_mon_vals no callback set 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f678e52c640 10 client.4152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f678e52c640 10 monclient: _renew_subs 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f678e52c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f678e52c640 1 -- 172.21.15.177:0/1697322656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67881d7b00 con 0x7f67880a8b20 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.764+0000 7f678e52c640 1 librados: init done 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.765+0000 7f678e52c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.765+0000 7f678e52c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:02.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.765+0000 7f678e52c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:02.755 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.765+0000 7f678e52c640 1 -- 172.21.15.177:0/1697322656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678804e970 con 0x7f67880a8b20 2024-01-21T03:19:02.755 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.766+0000 7f6784ff9640 1 -- 172.21.15.177:0/1697322656 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f677c005d10 con 0x7f67880a8b20 2024-01-21T03:19:02.755 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.766+0000 7f6784ff9640 1 -- 172.21.15.177:0/1697322656 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f677c04eb40 con 0x7f67880a8b20 2024-01-21T03:19:02.755 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.766+0000 7f6784ff9640 10 client.4152.objecter ms_dispatch 0x7f6788000b90 osd_map(9..9 src has 1..9) v4 2024-01-21T03:19:02.755 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.766+0000 7f6784ff9640 3 client.4152.objecter handle_osd_map got epochs [9,9] > 0 2024-01-21T03:19:02.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.766+0000 7f6784ff9640 3 client.4152.objecter handle_osd_map decoding full epoch 9 2024-01-21T03:19:02.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.766+0000 7f6784ff9640 20 client.4152.objecter dump_active .. 0 homeless 2024-01-21T03:19:02.860 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.873+0000 7f6784ff9640 1 -- 172.21.15.177:0/1697322656 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f677c04ed00 con 0x7f67880a8b20 2024-01-21T03:19:02.860 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.873+0000 7f6784ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:02.860 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.873+0000 7f6784ff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:02.864 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:02.878+0000 7f7ae9c13040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-21T03:19:02.921 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:02.934+0000 7f7ae9c13040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-21T03:19:02.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.945+0000 7f678e52c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e", "id": 7}] 2024-01-21T03:19:02.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.945+0000 7f678e52c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e", "id": 7}] 2024-01-21T03:19:02.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.945+0000 7f678e52c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:02.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.945+0000 7f678e52c640 1 -- 172.21.15.177:0/1697322656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e", "id": 7} v 0) v1 -- 0x7f6788072860 con 0x7f67880a8b20 2024-01-21T03:19:02.981 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.993+0000 7f6784ff9640 1 -- 172.21.15.177:0/1697322656 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f677c04e4e0 con 0x7f67880a8b20 2024-01-21T03:19:02.981 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.994+0000 7f6784ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e", "id": 7}] 2024-01-21T03:19:02.981 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.994+0000 7f6784ff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:02.981 INFO:teuthology.orchestra.run.smithi177.stdout:7 2024-01-21T03:19:02.983 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 10 librados: watch_flush enter 2024-01-21T03:19:02.983 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 10 librados: watch_flush exit 2024-01-21T03:19:02.983 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 20 client.4152.objecter shutdown clearing up homeless session... 2024-01-21T03:19:02.983 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 10 client.4152.objecter successfully canceled tick 2024-01-21T03:19:02.983 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 10 monclient: shutdown 2024-01-21T03:19:02.983 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:02.984 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 1 -- 172.21.15.177:0/1697322656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67880a8b20 msgr2=0x7f67880abc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:02.984 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 1 --2- 172.21.15.177:0/1697322656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67880a8b20 0x7f67880abc60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f677c0097a0 tx=0x7f677c0046c0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.984 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 1 -- 172.21.15.177:0/1697322656 shutdown_connections 2024-01-21T03:19:02.984 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 1 --2- 172.21.15.177:0/1697322656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67880b2310 0x7f67881d75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.984 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 1 --2- 172.21.15.177:0/1697322656 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f67880a9d10 0x7f67880ac8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.984 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 1 --2- 172.21.15.177:0/1697322656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67880a8b20 0x7f67880abc60 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:02.984 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 1 -- 172.21.15.177:0/1697322656 >> 172.21.15.177:0/1697322656 conn(0x7f6788066fb0 msgr2=0x7f678813ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:02.984 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 1 -- 172.21.15.177:0/1697322656 shutdown_connections 2024-01-21T03:19:02.984 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 1 -- 172.21.15.177:0/1697322656 wait complete. 2024-01-21T03:19:02.984 INFO:teuthology.orchestra.run.smithi177.stderr:2024-01-21T03:19:02.995+0000 7f678e52c640 1 librados: shutdown 2024-01-21T03:19:02.996 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T03:19:02.996 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-21T03:19:02.997 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-01-21T03:19:02.999 INFO:tasks.ceph.osd.0:Started 2024-01-21T03:19:02.999 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T03:19:02.999 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-21T03:19:02.999 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-01-21T03:19:03.001 INFO:tasks.ceph.osd.1:Started 2024-01-21T03:19:03.001 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T03:19:03.001 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-21T03:19:03.002 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-01-21T03:19:03.003 INFO:tasks.ceph.osd.2:Started 2024-01-21T03:19:03.004 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T03:19:03.004 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-21T03:19:03.004 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-01-21T03:19:03.006 INFO:tasks.ceph.osd.3:Started 2024-01-21T03:19:03.006 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T03:19:03.006 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-21T03:19:03.006 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-01-21T03:19:03.007 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:03.019+0000 7f1ba1a43040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-21T03:19:03.009 INFO:tasks.ceph.osd.4:Started 2024-01-21T03:19:03.009 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T03:19:03.009 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-21T03:19:03.009 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-01-21T03:19:03.011 INFO:tasks.ceph.osd.5:Started 2024-01-21T03:19:03.011 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T03:19:03.011 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-21T03:19:03.011 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-01-21T03:19:03.013 INFO:tasks.ceph.osd.6:Started 2024-01-21T03:19:03.013 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T03:19:03.014 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-21T03:19:03.014 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-01-21T03:19:03.016 INFO:tasks.ceph.osd.7:Started 2024-01-21T03:19:03.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-21T03:19:03.043 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:03.057+0000 7f7ae9c13040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-21T03:19:03.069 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:03.082+0000 7f1ba1a43040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-21T03:19:03.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.160+0000 7f9753b1d640 10 monclient: get_monmap_and_config 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.160+0000 7f9753b1d640 10 monclient: build_initial_monmap 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.161+0000 7f9753b1d640 10 monclient: monmap: 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:19:03.161967+0000 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:19:03.161967+0000 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:19:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.161+0000 7f9753b1d640 1 Processor -- start 2024-01-21T03:19:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.161+0000 7f9753b1d640 1 -- start start 2024-01-21T03:19:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.161+0000 7f9753b1d640 10 monclient: init 2024-01-21T03:19:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-01-21T03:19:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f974c0a8a40 0x7f974c145b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f974c0a8a40 addr v2:172.21.15.7:3300/0 2024-01-21T03:19:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f974c1460a0 addr v1:172.21.15.7:6789/0 2024-01-21T03:19:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c13f8b0 0x7f974c14c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f974c13f8b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974c0aa9b0 con 0x7f974c1460a0 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974c0aa7d0 con 0x7f974c0a8a40 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0aac90 con 0x7f974c13f8b0 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.162+0000 7f9753b1d640 10 monclient(hunting): authenticate will time out at 1289.478406s 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.163+0000 7f97520ba640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c13f8b0 0x7f974c14c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.164+0000 7f97520ba640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c13f8b0 0x7f974c14c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:53746/0 (socket says 172.21.15.7:53746) 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.164+0000 7f97520ba640 1 -- 172.21.15.7:0/287434806 learned_addr learned my addr 172.21.15.7:0/287434806 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.164+0000 7f97520ba640 10 monclient(hunting): get_auth_request con 0x7f974c13f8b0 auth_method 0 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.164+0000 7f97520ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.164+0000 7f97520ba640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.164+0000 7f97520ba640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.165+0000 7f97520ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.165+0000 7f97520ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.165+0000 7f97520ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.167+0000 7f97520ba640 10 monclient(hunting): handle_auth_done global_id 4142 payload 274 2024-01-21T03:19:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.167+0000 7f97520ba640 1 -- 172.21.15.7:0/287434806 >> v2:172.21.15.7:3300/0 conn(0x7f974c0a8a40 msgr2=0x7f974c145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.167+0000 7f97520ba640 1 --2- 172.21.15.7:0/287434806 >> v2:172.21.15.7:3300/0 conn(0x7f974c0a8a40 0x7f974c145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.167+0000 7f97520ba640 1 -- 172.21.15.7:0/287434806 >> v1:172.21.15.7:6789/0 conn(0x7f974c1460a0 legacy=0x7f974c13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.167+0000 7f97520ba640 10 monclient: _finish_hunting 0 2024-01-21T03:19:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.167+0000 7f97520ba640 1 monclient: found mon.noname-b 2024-01-21T03:19:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.167+0000 7f97520ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.167+0000 7f97520ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:19:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.167+0000 7f97520ba640 1 -- 172.21.15.7:0/287434806 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c0aaf70 con 0x7f974c13f8b0 2024-01-21T03:19:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.169+0000 7f97520ba640 1 --2- 172.21.15.7:0/287434806 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c13f8b0 0x7f974c14c7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9744009e50 tx=0x7f974402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8670e0549f6e6b5f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:03.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.173+0000 7f97508b7640 1 -- 172.21.15.7:0/287434806 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974402cb00 con 0x7f974c13f8b0 2024-01-21T03:19:03.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.173+0000 7f97508b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:03.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.173+0000 7f97508b7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:03.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.173+0000 7f97508b7640 10 monclient: dump: 2024-01-21T03:19:03.166 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:19:03.166 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:03.166 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:03.166 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:03.166 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:03.166 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.173+0000 7f97508b7640 10 monclient: _finish_auth 0 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.173+0000 7f97508b7640 10 monclient: _check_auth_tickets 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.173+0000 7f97508b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.173+0000 7f9753b1d640 5 monclient: authenticate success, global_id 4142 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.173+0000 7f9753b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.173+0000 7f97508b7640 1 -- 172.21.15.7:0/287434806 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974402cc60 con 0x7f974c13f8b0 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.173+0000 7f97508b7640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.173+0000 7f9753b1d640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.173+0000 7f9753b1d640 4 set_mon_vals no callback set 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f97508b7640 1 -- 172.21.15.7:0/287434806 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97440343f0 con 0x7f974c13f8b0 2024-01-21T03:19:03.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f97508b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f97508b7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f97508b7640 10 monclient: dump: 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f9753b1d640 10 monclient: shutdown 2024-01-21T03:19:03.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f9753b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f9753b1d640 1 -- 172.21.15.7:0/287434806 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c13f8b0 msgr2=0x7f974c14c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f9753b1d640 1 --2- 172.21.15.7:0/287434806 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c13f8b0 0x7f974c14c7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9744009e50 tx=0x7f974402bf60 comp rx=0 tx=0).stop 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f9753b1d640 1 -- 172.21.15.7:0/287434806 shutdown_connections 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f9753b1d640 1 --2- 172.21.15.7:0/287434806 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c13f8b0 0x7f974c14c7e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f9753b1d640 1 --2- 172.21.15.7:0/287434806 >> v2:172.21.15.7:3300/0 conn(0x7f974c0a8a40 0x7f974c145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f9753b1d640 1 -- 172.21.15.7:0/287434806 >> 172.21.15.7:0/287434806 conn(0x7f974c066f50 msgr2=0x7f974c0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f9753b1d640 1 -- 172.21.15.7:0/287434806 shutdown_connections 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.174+0000 7f9753b1d640 1 -- 172.21.15.7:0/287434806 wait complete. 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.176+0000 7f9753b1d640 10 monclient: build_initial_monmap 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.176+0000 7f9753b1d640 10 monclient: monmap: 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:19:03.169 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.176+0000 7f9753b1d640 1 librados: starting msgr at 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.176+0000 7f9753b1d640 1 librados: starting objecter 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.176+0000 7f9753b1d640 1 Processor -- start 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.176+0000 7f9753b1d640 1 -- start start 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.176+0000 7f9753b1d640 1 librados: setting wanted keys 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.176+0000 7f9753b1d640 1 librados: calling monclient init 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.176+0000 7f9753b1d640 10 monclient: init 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T03:19:03.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c0a8a40 0x7f974c1365d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:03.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f974c0a8a40 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:03.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f974c13f8b0 0x7f974c141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:03.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f974c13f8b0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:03.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f974c1460a0 0x7f974c13fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:03.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f974c1460a0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:03.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:03.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c142300 con 0x7f974c13f8b0 2024-01-21T03:19:03.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:03.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0b6630 con 0x7f974c1460a0 2024-01-21T03:19:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974c1428d0 con 0x7f974c0a8a40 2024-01-21T03:19:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.177+0000 7f9753b1d640 10 monclient(hunting): authenticate will time out at 1289.493257s 2024-01-21T03:19:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.178+0000 7f97518b9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c0a8a40 0x7f974c1365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.178+0000 7f97518b9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c0a8a40 0x7f974c1365d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:53758/0 (socket says 172.21.15.7:53758) 2024-01-21T03:19:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.178+0000 7f97518b9640 1 -- 172.21.15.7:0/3525402891 learned_addr learned my addr 172.21.15.7:0/3525402891 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.178+0000 7f97518b9640 10 monclient(hunting): get_auth_request con 0x7f974c0a8a40 auth_method 0 2024-01-21T03:19:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.178+0000 7f97518b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.178+0000 7f97518b9640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.178+0000 7f97518b9640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.178+0000 7f97520ba640 1 --2- 172.21.15.7:0/3525402891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f974c1460a0 0x7f974c13fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.179+0000 7f97520ba640 10 monclient(hunting): get_auth_request con 0x7f974c1460a0 auth_method 0 2024-01-21T03:19:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.179+0000 7f97520ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.179+0000 7f97520ba640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.179+0000 7f97520ba640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.179+0000 7f97518b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.179+0000 7f97518b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.179+0000 7f97518b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.179+0000 7f97520ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.179+0000 7f97520ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.179+0000 7f97520ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:03.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.182+0000 7f97518b9640 10 monclient(hunting): handle_auth_done global_id 4145 payload 824 2024-01-21T03:19:03.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.182+0000 7f97518b9640 1 -- 172.21.15.7:0/3525402891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f974c1460a0 msgr2=0x7f974c13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:03.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.182+0000 7f97518b9640 1 --2- 172.21.15.7:0/3525402891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f974c1460a0 0x7f974c13fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:03.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.182+0000 7f97518b9640 1 -- 172.21.15.7:0/3525402891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f974c13f8b0 msgr2=0x7f974c141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:03.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.182+0000 7f97518b9640 1 --2- 172.21.15.7:0/3525402891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f974c13f8b0 0x7f974c141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:03.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.182+0000 7f97518b9640 10 monclient: _finish_hunting 0 2024-01-21T03:19:03.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.182+0000 7f97518b9640 1 monclient: found mon.noname-c 2024-01-21T03:19:03.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.182+0000 7f97518b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:03.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.182+0000 7f97518b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:19:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.182+0000 7f97518b9640 1 -- 172.21.15.7:0/3525402891 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9744009b30 con 0x7f974c0a8a40 2024-01-21T03:19:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.184+0000 7f97518b9640 1 --2- 172.21.15.7:0/3525402891 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c0a8a40 0x7f974c1365d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f974800b500 tx=0x7f974800b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.184+0000 7f97427fc640 10 client.?.objecter ms_handle_connect 0x7f974c0a8a40 2024-01-21T03:19:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.184+0000 7f97427fc640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f97427fc640 1 -- 172.21.15.7:0/3525402891 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748010870 con 0x7f974c0a8a40 2024-01-21T03:19:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f97427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f97427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:19:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f97427fc640 10 monclient: dump: 2024-01-21T03:19:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:19:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:03.180 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:19:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f97427fc640 10 monclient: _finish_auth 0 2024-01-21T03:19:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f97427fc640 10 monclient: _check_auth_tickets 2024-01-21T03:19:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f97427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f97427fc640 1 -- 172.21.15.7:0/3525402891 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97480109d0 con 0x7f974c0a8a40 2024-01-21T03:19:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f97427fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f97427fc640 1 -- 172.21.15.7:0/3525402891 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748010d80 con 0x7f974c0a8a40 2024-01-21T03:19:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f97427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f97427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:19:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f97427fc640 10 monclient: dump: 2024-01-21T03:19:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:19:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:03.181 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f9753b1d640 5 monclient: authenticate success, global_id 4145 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f9743fff640 4 set_mon_vals no callback set 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f9753b1d640 10 monclient: _renew_subs 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f9753b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.188+0000 7f9753b1d640 1 -- 172.21.15.7:0/3525402891 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c0b6880 con 0x7f974c0a8a40 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.189+0000 7f9753b1d640 10 client.4145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.189+0000 7f9753b1d640 10 monclient: _renew_subs 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.189+0000 7f9753b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.189+0000 7f9753b1d640 1 -- 172.21.15.7:0/3525402891 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974c140520 con 0x7f974c0a8a40 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.189+0000 7f9753b1d640 1 librados: init done 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.190+0000 7f9753b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.190+0000 7f9753b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.190+0000 7f9753b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:19:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.190+0000 7f9753b1d640 1 -- 172.21.15.7:0/3525402891 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974c04e970 con 0x7f974c0a8a40 2024-01-21T03:19:03.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.194+0000 7f97427fc640 1 -- 172.21.15.7:0/3525402891 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f9748023480 con 0x7f974c0a8a40 2024-01-21T03:19:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.198+0000 7f97427fc640 1 -- 172.21.15.7:0/3525402891 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f974800f3c0 con 0x7f974c0a8a40 2024-01-21T03:19:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.198+0000 7f97427fc640 10 client.4145.objecter ms_dispatch 0x7f974c000b90 osd_map(10..10 src has 1..10) v4 2024-01-21T03:19:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.198+0000 7f97427fc640 3 client.4145.objecter handle_osd_map got epochs [10,10] > 0 2024-01-21T03:19:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.198+0000 7f97427fc640 3 client.4145.objecter handle_osd_map decoding full epoch 10 2024-01-21T03:19:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.198+0000 7f97427fc640 20 client.4145.objecter dump_active .. 0 homeless 2024-01-21T03:19:03.184 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-01-21T03:19:03.197+0000 7f1ba1a43040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-21T03:19:03.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.378+0000 7f97427fc640 1 -- 172.21.15.7:0/3525402891 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f974801e060 con 0x7f974c0a8a40 2024-01-21T03:19:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.378+0000 7f97427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.378+0000 7f97427fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:03.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.503+0000 7f9753b1d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:03.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.503+0000 7f9753b1d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:03.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.503+0000 7f9753b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:19:03.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.503+0000 7f9753b1d640 1 -- 172.21.15.7:0/3525402891 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f974c072770 con 0x7f974c0a8a40 2024-01-21T03:19:03.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.619+0000 7f97427fc640 1 -- 172.21.15.7:0/3525402891 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mgrmap(e 2) v1 ==== 44620+0+0 (secure 0 0 0) 0x7f9748023730 con 0x7f974c0a8a40 2024-01-21T03:19:03.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.700+0000 7f97427fc640 1 -- 172.21.15.7:0/3525402891 <== mon.2 v2:172.21.15.7:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f9748019720 con 0x7f974c0a8a40 2024-01-21T03:19:03.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.700+0000 7f97427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:03.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.700+0000 7f97427fc640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:19:03.688 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":10,"fsid":"8de9e7dd-e982-4048-83dd-ee52350dd56d","created":"2024-01-21T03:18:57.607470+0000","modified":"2024-01-21T03:19:02.952334+0000","last_up_change":"0.000000","last_in_change":"2024-01-21T03:19:02.952334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f4767aa-192e-4ec9-addf-a51c0872da7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"066d7378-561f-4e74-b97b-2ead2fa2625e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f219649b-4d50-40a8-b74e-5a7f16b3c8c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6379a402-760a-4f07-9491-0918752a18fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dc3602e1-b59e-4489-9882-40c914e2a0d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d42afb34-e1ca-4b21-81e9-78d1c8ea454e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1aef2850-8b41-4d39-859e-ed9c4df434b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T03:19:03.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.702+0000 7f9753b1d640 10 librados: watch_flush enter 2024-01-21T03:19:03.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.702+0000 7f9753b1d640 10 librados: watch_flush exit 2024-01-21T03:19:03.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.702+0000 7f9753b1d640 20 client.4145.objecter shutdown clearing up homeless session... 2024-01-21T03:19:03.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.702+0000 7f9753b1d640 10 client.4145.objecter successfully canceled tick 2024-01-21T03:19:03.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.702+0000 7f9753b1d640 10 monclient: shutdown 2024-01-21T03:19:03.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.702+0000 7f9753b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:03.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.702+0000 7f9753b1d640 1 -- 172.21.15.7:0/3525402891 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c0a8a40 msgr2=0x7f974c1365d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:03.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.702+0000 7f9753b1d640 1 --2- 172.21.15.7:0/3525402891 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c0a8a40 0x7f974c1365d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f974800b500 tx=0x7f974800b9c0 comp rx=0 tx=0).stop 2024-01-21T03:19:03.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.702+0000 7f9753b1d640 1 -- 172.21.15.7:0/3525402891 shutdown_connections 2024-01-21T03:19:03.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.702+0000 7f9753b1d640 1 --2- 172.21.15.7:0/3525402891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f974c1460a0 0x7f974c13fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:03.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.702+0000 7f9753b1d640 1 --2- 172.21.15.7:0/3525402891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f974c13f8b0 0x7f974c141dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:03.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.702+0000 7f9753b1d640 1 --2- 172.21.15.7:0/3525402891 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f974c0a8a40 0x7f974c1365d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:03.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.702+0000 7f9753b1d640 1 -- 172.21.15.7:0/3525402891 >> 172.21.15.7:0/3525402891 conn(0x7f974c066f50 msgr2=0x7f974c0b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:03.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.703+0000 7f9753b1d640 1 -- 172.21.15.7:0/3525402891 shutdown_connections 2024-01-21T03:19:03.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.703+0000 7f9753b1d640 1 -- 172.21.15.7:0/3525402891 wait complete. 2024-01-21T03:19:03.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:03.703+0000 7f9753b1d640 1 librados: shutdown 2024-01-21T03:19:03.704 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-21T03:19:03.704 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-21T03:19:04.510 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T03:19:04.663 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.677+0000 7f2bcb29b640 10 monclient: get_monmap_and_config 2024-01-21T03:19:04.664 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.677+0000 7f2bcb29b640 10 monclient: build_initial_monmap 2024-01-21T03:19:04.664 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.677+0000 7f2bcb29b640 10 monclient: monmap: 2024-01-21T03:19:04.664 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:04.664 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:04.664 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:19:04.678508+0000 2024-01-21T03:19:04.664 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:19:04.678508+0000 2024-01-21T03:19:04.664 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:04.664 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:04.664 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:04.664 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:04.664 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:04.665 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:04.665 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:04.665 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:04.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.678+0000 7f2bcb29b640 1 Processor -- start 2024-01-21T03:19:04.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.678+0000 7f2bcb29b640 1 -- start start 2024-01-21T03:19:04.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.678+0000 7f2bcb29b640 10 monclient: init 2024-01-21T03:19:04.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:04.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-01-21T03:19:04.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2bc40a8a60 0x7f2bc413f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:04.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 10 monclient(hunting): picked mon.noname-a con 0x7f2bc40a8a60 addr v2:172.21.15.7:3300/0 2024-01-21T03:19:04.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2bc413f840 addr v1:172.21.15.177:6789/0 2024-01-21T03:19:04.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2bc41428e0 0x7f2bc414c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:04.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 10 monclient(hunting): picked mon.noname-b con 0x7f2bc41428e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:04.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc40b1de0 con 0x7f2bc413f840 2024-01-21T03:19:04.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:04.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc40a8e40 con 0x7f2bc40a8a60 2024-01-21T03:19:04.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:04.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc413fc70 con 0x7f2bc41428e0 2024-01-21T03:19:04.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:04.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.679+0000 7f2bcb29b640 10 monclient(hunting): authenticate will time out at 1290.995476s 2024-01-21T03:19:04.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.680+0000 7f2bc9838640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2bc41428e0 0x7f2bc414c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:04.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.680+0000 7f2bc8836640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f2bc413f840 0x7f2bc41421d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:52050/0 (socket says 172.21.15.7:52050) 2024-01-21T03:19:04.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.680+0000 7f2bc8836640 1 -- 172.21.15.7:0/3659867938 learned_addr learned my addr 172.21.15.7:0/3659867938 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:19:04.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.681+0000 7f2bc9838640 10 monclient(hunting): get_auth_request con 0x7f2bc41428e0 auth_method 0 2024-01-21T03:19:04.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.681+0000 7f2bc9838640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:04.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.681+0000 7f2bc9838640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:04.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.681+0000 7f2bc9838640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:04.672 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.681+0000 7f2bc9838640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:04.672 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.681+0000 7f2bc9838640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:04.672 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.681+0000 7f2bc9838640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:04.672 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.684+0000 7f2bc9838640 10 monclient(hunting): handle_auth_done global_id 4163 payload 274 2024-01-21T03:19:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.684+0000 7f2bc9838640 1 -- 172.21.15.7:0/3659867938 >> v2:172.21.15.7:3300/0 conn(0x7f2bc40a8a60 msgr2=0x7f2bc413f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.684+0000 7f2bc9838640 1 --2- 172.21.15.7:0/3659867938 >> v2:172.21.15.7:3300/0 conn(0x7f2bc40a8a60 0x7f2bc413f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.684+0000 7f2bc9838640 1 -- 172.21.15.7:0/3659867938 >> v1:172.21.15.177:6789/0 conn(0x7f2bc413f840 legacy=0x7f2bc41421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.684+0000 7f2bc9838640 10 monclient: _finish_hunting 0 2024-01-21T03:19:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.684+0000 7f2bc9838640 1 monclient: found mon.noname-b 2024-01-21T03:19:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.684+0000 7f2bc9838640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.684+0000 7f2bc9838640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:19:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.684+0000 7f2bc9838640 1 -- 172.21.15.7:0/3659867938 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc40aa480 con 0x7f2bc41428e0 2024-01-21T03:19:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.687+0000 7f2bc9838640 1 --2- 172.21.15.7:0/3659867938 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2bc41428e0 0x7f2bc414c930 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009d90 tx=0x7f2bbc02be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac86188cfd55fdea server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.690+0000 7f2bbbfff640 1 -- 172.21.15.7:0/3659867938 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc02cbe0 con 0x7f2bc41428e0 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.690+0000 7f2bbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.690+0000 7f2bbbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.690+0000 7f2bbbfff640 10 monclient: dump: 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:04.677 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:04.678 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.690+0000 7f2bbbfff640 10 monclient: _finish_auth 0 2024-01-21T03:19:04.678 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.690+0000 7f2bbbfff640 10 monclient: _check_auth_tickets 2024-01-21T03:19:04.678 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.690+0000 7f2bbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:04.678 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.691+0000 7f2bcb29b640 5 monclient: authenticate success, global_id 4163 2024-01-21T03:19:04.678 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.691+0000 7f2bcb29b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:19:04.679 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.693+0000 7f2bbbfff640 1 -- 172.21.15.7:0/3659867938 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc02cd40 con 0x7f2bc41428e0 2024-01-21T03:19:04.679 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.693+0000 7f2bbbfff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:04.679 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.693+0000 7f2bbbfff640 1 -- 172.21.15.7:0/3659867938 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc034660 con 0x7f2bc41428e0 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.693+0000 7f2bcb29b640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.693+0000 7f2bcb29b640 4 set_mon_vals no callback set 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.693+0000 7f2bbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.693+0000 7f2bbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.693+0000 7f2bbbfff640 10 monclient: dump: 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:04.680 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:04.681 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:04.684 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.694+0000 7f2bcb29b640 10 monclient: shutdown 2024-01-21T03:19:04.684 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.694+0000 7f2bcb29b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:04.684 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.694+0000 7f2bcb29b640 1 -- 172.21.15.7:0/3659867938 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2bc41428e0 msgr2=0x7f2bc414c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:04.684 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.694+0000 7f2bcb29b640 1 --2- 172.21.15.7:0/3659867938 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2bc41428e0 0x7f2bc414c930 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009d90 tx=0x7f2bbc02be30 comp rx=0 tx=0).stop 2024-01-21T03:19:04.684 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.694+0000 7f2bcb29b640 1 -- 172.21.15.7:0/3659867938 shutdown_connections 2024-01-21T03:19:04.684 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.694+0000 7f2bcb29b640 1 --2- 172.21.15.7:0/3659867938 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2bc41428e0 0x7f2bc414c930 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:04.684 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.694+0000 7f2bcb29b640 1 --2- 172.21.15.7:0/3659867938 >> v2:172.21.15.7:3300/0 conn(0x7f2bc40a8a60 0x7f2bc413f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.694+0000 7f2bcb29b640 1 -- 172.21.15.7:0/3659867938 >> 172.21.15.7:0/3659867938 conn(0x7f2bc40670b0 msgr2=0x7f2bc40b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.694+0000 7f2bcb29b640 1 -- 172.21.15.7:0/3659867938 shutdown_connections 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.694+0000 7f2bcb29b640 1 -- 172.21.15.7:0/3659867938 wait complete. 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.695+0000 7f2bcb29b640 10 monclient: build_initial_monmap 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.695+0000 7f2bcb29b640 10 monclient: monmap: 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:04.685 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.696+0000 7f2bcb29b640 1 librados: starting msgr at 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.696+0000 7f2bcb29b640 1 librados: starting objecter 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.696+0000 7f2bcb29b640 1 Processor -- start 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.696+0000 7f2bcb29b640 1 -- start start 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.696+0000 7f2bcb29b640 1 librados: setting wanted keys 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.696+0000 7f2bcb29b640 1 librados: calling monclient init 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.696+0000 7f2bcb29b640 10 monclient: init 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.697+0000 7f2bcb29b640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.697+0000 7f2bcb29b640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.697+0000 7f2bcb29b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2bc413f840 0x7f2bc41460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.697+0000 7f2bcb29b640 10 monclient(hunting): picked mon.noname-c con 0x7f2bc413f840 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.697+0000 7f2bcb29b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bc41428e0 0x7f2bc4143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.697+0000 7f2bcb29b640 10 monclient(hunting): picked mon.noname-b con 0x7f2bc41428e0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.697+0000 7f2bcb29b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bc4144350 0x7f2bc40b5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.697+0000 7f2bcb29b640 10 monclient(hunting): picked mon.noname-a con 0x7f2bc4144350 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:04.686 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.697+0000 7f2bcb29b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:04.687 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.697+0000 7f2bcb29b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc4143680 con 0x7f2bc4144350 2024-01-21T03:19:04.687 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.697+0000 7f2bcb29b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:04.687 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.697+0000 7f2bcb29b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc41433a0 con 0x7f2bc41428e0 2024-01-21T03:19:04.687 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.698+0000 7f2bcb29b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:04.687 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.698+0000 7f2bcb29b640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc41453c0 con 0x7f2bc413f840 2024-01-21T03:19:04.687 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.698+0000 7f2bcb29b640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:04.687 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.698+0000 7f2bcb29b640 10 monclient(hunting): authenticate will time out at 1291.013958s 2024-01-21T03:19:04.688 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.701+0000 7f2bc9037640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2bc413f840 0x7f2bc41460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:04.688 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.702+0000 7f2bc9037640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2bc413f840 0x7f2bc41460a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:53834/0 (socket says 172.21.15.7:53834) 2024-01-21T03:19:04.688 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.702+0000 7f2bc9037640 1 -- 172.21.15.7:0/2468923434 learned_addr learned my addr 172.21.15.7:0/2468923434 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:04.688 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.702+0000 7f2bc9037640 10 monclient(hunting): get_auth_request con 0x7f2bc413f840 auth_method 0 2024-01-21T03:19:04.688 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.702+0000 7f2bc9037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:04.688 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.702+0000 7f2bc9037640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:04.688 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.702+0000 7f2bc9037640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:04.689 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.703+0000 7f2bc9037640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:04.689 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.703+0000 7f2bc9037640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:04.689 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.703+0000 7f2bc9037640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:04.690 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.704+0000 7f2bc8836640 1 --2- 172.21.15.7:0/2468923434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bc41428e0 0x7f2bc4143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:04.696 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.709+0000 7f2bc9037640 10 monclient(hunting): handle_auth_done global_id 4166 payload 824 2024-01-21T03:19:04.697 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.709+0000 7f2bc9037640 1 -- 172.21.15.7:0/2468923434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bc41428e0 msgr2=0x7f2bc4143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:04.697 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.709+0000 7f2bc9037640 1 --2- 172.21.15.7:0/2468923434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bc41428e0 0x7f2bc4143e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:04.697 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.709+0000 7f2bc9037640 1 -- 172.21.15.7:0/2468923434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bc4144350 msgr2=0x7f2bc40b5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:04.697 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.709+0000 7f2bc9037640 1 --2- 172.21.15.7:0/2468923434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bc4144350 0x7f2bc40b5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:04.697 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.709+0000 7f2bc9037640 10 monclient: _finish_hunting 0 2024-01-21T03:19:04.697 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.709+0000 7f2bc9037640 1 monclient: found mon.noname-c 2024-01-21T03:19:04.697 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.709+0000 7f2bc9037640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:04.697 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.709+0000 7f2bc9037640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:19:04.697 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.709+0000 7f2bc9037640 1 -- 172.21.15.7:0/2468923434 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc009a70 con 0x7f2bc413f840 2024-01-21T03:19:04.698 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.712+0000 7f2bc9037640 1 --2- 172.21.15.7:0/2468923434 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2bc413f840 0x7f2bc41460a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000b440 tx=0x7f2bc000b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:04.698 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.712+0000 7f2bb9ffb640 10 client.?.objecter ms_handle_connect 0x7f2bc413f840 2024-01-21T03:19:04.698 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.712+0000 7f2bb9ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:04.704 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.717+0000 7f2bb9ffb640 1 -- 172.21.15.7:0/2468923434 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0011890 con 0x7f2bc413f840 2024-01-21T03:19:04.704 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.717+0000 7f2bb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:04.704 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.718+0000 7f2bb9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:19:04.704 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.718+0000 7f2bb9ffb640 10 monclient: dump: 2024-01-21T03:19:04.704 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:04.704 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:04.704 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:04.704 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:04.704 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:04.704 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:04.704 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:04.704 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.718+0000 7f2bb9ffb640 10 monclient: _finish_auth 0 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.718+0000 7f2bb9ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.718+0000 7f2bb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.718+0000 7f2bb9ffb640 1 -- 172.21.15.7:0/2468923434 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc00119f0 con 0x7f2bc413f840 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.718+0000 7f2bcb29b640 5 monclient: authenticate success, global_id 4166 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.718+0000 7f2bb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.718+0000 7f2bb9ffb640 1 -- 172.21.15.7:0/2468923434 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0011da0 con 0x7f2bc413f840 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.718+0000 7f2bb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.718+0000 7f2bb9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.718+0000 7f2bb9ffb640 10 monclient: dump: 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:04.705 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:04.706 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:04.706 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:04.706 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:04.706 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:04.706 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:04.706 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:04.706 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:04.706 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:04.706 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.718+0000 7f2bbb7fe640 4 set_mon_vals no callback set 2024-01-21T03:19:04.707 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.719+0000 7f2bcb29b640 10 monclient: _renew_subs 2024-01-21T03:19:04.707 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.719+0000 7f2bcb29b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:19:04.707 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.719+0000 7f2bcb29b640 1 -- 172.21.15.7:0/2468923434 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc4145630 con 0x7f2bc413f840 2024-01-21T03:19:04.708 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.719+0000 7f2bcb29b640 10 client.4166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:04.708 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.719+0000 7f2bcb29b640 10 monclient: _renew_subs 2024-01-21T03:19:04.708 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.719+0000 7f2bcb29b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:19:04.708 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.719+0000 7f2bcb29b640 1 -- 172.21.15.7:0/2468923434 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc4143b10 con 0x7f2bc413f840 2024-01-21T03:19:04.708 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.719+0000 7f2bcb29b640 1 librados: init done 2024-01-21T03:19:04.708 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.720+0000 7f2bcb29b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:04.708 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.720+0000 7f2bcb29b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:04.708 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.720+0000 7f2bcb29b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:19:04.708 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.720+0000 7f2bcb29b640 1 -- 172.21.15.7:0/2468923434 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc406b500 con 0x7f2bc413f840 2024-01-21T03:19:04.725 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.739+0000 7f2bb9ffb640 1 -- 172.21.15.7:0/2468923434 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 2) v1 ==== 44620+0+0 (secure 0 0 0) 0x7f2bc001a400 con 0x7f2bc413f840 2024-01-21T03:19:04.909 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.923+0000 7f2bb9ffb640 1 -- 172.21.15.7:0/2468923434 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f2bc001a6b0 con 0x7f2bc413f840 2024-01-21T03:19:04.910 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.923+0000 7f2bb9ffb640 10 client.4166.objecter ms_dispatch 0x7f2bc4000b90 osd_map(10..10 src has 1..10) v4 2024-01-21T03:19:04.910 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.923+0000 7f2bb9ffb640 3 client.4166.objecter handle_osd_map got epochs [10,10] > 0 2024-01-21T03:19:04.910 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.923+0000 7f2bb9ffb640 3 client.4166.objecter handle_osd_map decoding full epoch 10 2024-01-21T03:19:04.910 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.923+0000 7f2bb9ffb640 20 client.4166.objecter dump_active .. 0 homeless 2024-01-21T03:19:04.910 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.923+0000 7f2bb9ffb640 1 -- 172.21.15.7:0/2468923434 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f2bc00103e0 con 0x7f2bc413f840 2024-01-21T03:19:04.910 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.923+0000 7f2bb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:04.910 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:04.923+0000 7f2bb9ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:05.027 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:05.041+0000 7f7a8ab5c640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-21T03:19:05.036 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.050+0000 7f2bcb29b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:05.036 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.050+0000 7f2bcb29b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:05.036 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.050+0000 7f2bcb29b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:19:05.036 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.050+0000 7f2bcb29b640 1 -- 172.21.15.7:0/2468923434 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bc40b4140 con 0x7f2bc413f840 2024-01-21T03:19:05.255 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.269+0000 7f2bb9ffb640 1 -- 172.21.15.7:0/2468923434 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mgrmap(e 3) v1 ==== 89036+0+0 (secure 0 0 0) 0x7f2bc001a8f0 con 0x7f2bc413f840 2024-01-21T03:19:05.256 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.269+0000 7f2bb9ffb640 1 --2- 172.21.15.7:0/2468923434 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f2ba806cd20 0x7f2ba806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:05.256 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.269+0000 7f2bc8836640 1 --2- 172.21.15.7:0/2468923434 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f2ba806cd20 0x7f2ba806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:05.256 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.269+0000 7f2bc8836640 10 monclient: get_auth_request con 0x7f2ba806cd20 auth_method 0 2024-01-21T03:19:05.256 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.270+0000 7f2bb9ffb640 10 client.4166.objecter ms_handle_connect 0x7f2ba806cd20 2024-01-21T03:19:05.256 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.270+0000 7f2bc8836640 1 --2- 172.21.15.7:0/2468923434 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f2ba806cd20 0x7f2ba806f110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4144fa0 tx=0x7f2bb4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:05.465 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-01-21T03:19:05.479+0000 7f7a8ab5c640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-21T03:19:05.499 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.513+0000 7f2bb9ffb640 1 -- 172.21.15.7:0/2468923434 <== mon.2 v2:172.21.15.7:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f2bc00578e0 con 0x7f2bc413f840 2024-01-21T03:19:05.499 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.513+0000 7f2bb9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:05.499 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.513+0000 7f2bb9ffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:05.499 INFO:teuthology.misc.health.smithi007.stdout: 2024-01-21T03:19:05.499 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"8de9e7dd-e982-4048-83dd-ee52350dd56d","created":"2024-01-21T03:18:57.607470+0000","modified":"2024-01-21T03:19:02.952334+0000","last_up_change":"0.000000","last_in_change":"2024-01-21T03:19:02.952334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f4767aa-192e-4ec9-addf-a51c0872da7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"066d7378-561f-4e74-b97b-2ead2fa2625e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f219649b-4d50-40a8-b74e-5a7f16b3c8c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6379a402-760a-4f07-9491-0918752a18fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dc3602e1-b59e-4489-9882-40c914e2a0d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d42afb34-e1ca-4b21-81e9-78d1c8ea454e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1aef2850-8b41-4d39-859e-ed9c4df434b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T03:19:05.501 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 10 librados: watch_flush enter 2024-01-21T03:19:05.501 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 10 librados: watch_flush exit 2024-01-21T03:19:05.501 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 20 client.4166.objecter shutdown clearing up homeless session... 2024-01-21T03:19:05.501 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 10 client.4166.objecter successfully canceled tick 2024-01-21T03:19:05.501 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 1 -- 172.21.15.7:0/2468923434 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f2ba806cd20 msgr2=0x7f2ba806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:05.501 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 1 --2- 172.21.15.7:0/2468923434 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f2ba806cd20 0x7f2ba806f110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4144fa0 tx=0x7f2bb4009040 comp rx=0 tx=0).stop 2024-01-21T03:19:05.501 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 10 monclient: shutdown 2024-01-21T03:19:05.501 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:05.501 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 1 -- 172.21.15.7:0/2468923434 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2bc413f840 msgr2=0x7f2bc41460a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:05.501 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 1 --2- 172.21.15.7:0/2468923434 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2bc413f840 0x7f2bc41460a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000b440 tx=0x7f2bc000b900 comp rx=0 tx=0).stop 2024-01-21T03:19:05.501 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 1 -- 172.21.15.7:0/2468923434 shutdown_connections 2024-01-21T03:19:05.501 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 1 --2- 172.21.15.7:0/2468923434 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f2ba806cd20 0x7f2ba806f110 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:05.502 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 1 --2- 172.21.15.7:0/2468923434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bc4144350 0x7f2bc40b5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:05.502 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 1 --2- 172.21.15.7:0/2468923434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bc41428e0 0x7f2bc4143e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:05.502 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 1 --2- 172.21.15.7:0/2468923434 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2bc413f840 0x7f2bc41460a0 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000b440 tx=0x7f2bc000b900 comp rx=0 tx=0).stop 2024-01-21T03:19:05.502 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.514+0000 7f2bcb29b640 1 -- 172.21.15.7:0/2468923434 >> 172.21.15.7:0/2468923434 conn(0x7f2bc40a8a60 msgr2=0x7f2bc40b4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:05.502 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.515+0000 7f2bcb29b640 1 -- 172.21.15.7:0/2468923434 shutdown_connections 2024-01-21T03:19:05.502 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.515+0000 7f2bcb29b640 1 -- 172.21.15.7:0/2468923434 wait complete. 2024-01-21T03:19:05.502 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:05.515+0000 7f2bcb29b640 1 librados: shutdown 2024-01-21T03:19:05.521 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-21T03:19:10.348 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:07.272 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:10.418 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:07.342 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:10.484 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:07.388 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:10.523 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:07.448 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:10.548 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:07.452 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:10.566 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:07.469 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:10.583 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:07.507 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:10.628 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:07.530 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:10.945 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:07.873 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:10.954 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:10.962+0000 5dc2300 -1 Falling back to public interface 2024-01-21T03:19:10.976 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:07.891 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:11.009 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:07.936 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:11.038 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:07.965 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:11.041 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:07.957 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:11.091 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:19:11.099+0000 5dc2300 -1 Falling back to public interface 2024-01-21T03:19:11.100 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:08.028 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:11.124 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:08.036 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:11.160 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:11.168+0000 5dc2300 -1 Falling back to public interface 2024-01-21T03:19:11.185 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-01-21T03:19:11.193+0000 5dc2300 -1 Falling back to public interface 2024-01-21T03:19:11.185 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:08.097 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:11.412 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:08.336 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:11.552 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:08.457 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:11.565 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:19:11.570+0000 5dc2300 -1 Falling back to public interface 2024-01-21T03:19:11.614 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:11.620+0000 5dc2300 -1 Falling back to public interface 2024-01-21T03:19:11.634 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:08.558 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:11.658 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:08.561 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:11.660 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:19:11.668+0000 5dc2300 -1 Falling back to public interface 2024-01-21T03:19:11.776 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:19:11.783+0000 5dc2300 -1 Falling back to public interface 2024-01-21T03:19:11.776 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:08.700 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:11.917 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:08.822 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.022 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:08.946 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.044 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:08.946 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.045 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:08.973 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.086 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:09.002 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.132 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:09.056 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.147 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:09.074 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.244 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:09.148 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.282 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:09.194 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.326 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T03:19:12.362 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:09.265 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.377 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:09.302 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.415 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.429+0000 7faf91289640 10 monclient: get_monmap_and_config 2024-01-21T03:19:12.416 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.429+0000 7faf91289640 10 monclient: build_initial_monmap 2024-01-21T03:19:12.416 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.429+0000 7faf91289640 10 monclient: monmap: 2024-01-21T03:19:12.416 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:12.416 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:12.416 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:19:12.430564+0000 2024-01-21T03:19:12.416 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:19:12.430564+0000 2024-01-21T03:19:12.416 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:12.416 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:12.416 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:12.416 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:12.416 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:12.416 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:12.416 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:12.417 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:12.419 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.430+0000 7faf91289640 1 Processor -- start 2024-01-21T03:19:12.419 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.430+0000 7faf91289640 1 -- start start 2024-01-21T03:19:12.419 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.430+0000 7faf91289640 10 monclient: init 2024-01-21T03:19:12.419 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:12.419 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-01-21T03:19:12.419 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faf8c0a8a60 0x7faf8c13f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:12.420 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 10 monclient(hunting): picked mon.noname-a con 0x7faf8c0a8a60 addr v2:172.21.15.7:3300/0 2024-01-21T03:19:12.420 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faf8c13f840 0x7faf8c1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:12.420 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 10 monclient(hunting): picked mon.noname-b con 0x7faf8c13f840 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:12.420 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf8c142710 addr v1:172.21.15.177:6789/0 2024-01-21T03:19:12.420 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8c0b1de0 con 0x7faf8c142710 2024-01-21T03:19:12.420 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:12.420 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf8c0a8e40 con 0x7faf8c0a8a60 2024-01-21T03:19:12.420 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:12.420 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c0aa480 con 0x7faf8c13f840 2024-01-21T03:19:12.420 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:12.420 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.431+0000 7faf91289640 10 monclient(hunting): authenticate will time out at 1298.747219s 2024-01-21T03:19:12.420 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.432+0000 7faf8b7fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7faf8c142710 0x7faf8c1455a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:43870/0 (socket says 172.21.15.7:43870) 2024-01-21T03:19:12.420 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.432+0000 7faf8b7fe640 1 -- 172.21.15.7:0/2529181268 learned_addr learned my addr 172.21.15.7:0/2529181268 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.432+0000 7faf8affd640 1 --2- 172.21.15.7:0/2529181268 >> v2:172.21.15.7:3300/0 conn(0x7faf8c0a8a60 0x7faf8c13f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.432+0000 7faf83fff640 1 --2- 172.21.15.7:0/2529181268 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faf8c13f840 0x7faf8c1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.432+0000 7faf83fff640 10 monclient(hunting): get_auth_request con 0x7faf8c13f840 auth_method 0 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.432+0000 7faf83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.432+0000 7faf83fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.432+0000 7faf83fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.432+0000 7faf8affd640 10 monclient(hunting): get_auth_request con 0x7faf8c0a8a60 auth_method 0 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.432+0000 7faf8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.432+0000 7faf8affd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.432+0000 7faf8affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.433+0000 7faf83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.433+0000 7faf83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.433+0000 7faf83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.433+0000 7faf8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:12.421 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.433+0000 7faf8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:12.422 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.433+0000 7faf8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.434+0000 7faf8a7fc640 1 -- 172.21.15.7:0/2529181268 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637047713 0 0) 0x7faf8c0b1de0 con 0x7faf8c142710 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.434+0000 7faf8a7fc640 10 monclient(hunting): _negotiate 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.434+0000 7faf8a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.434+0000 7faf8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.434+0000 7faf8a7fc640 10 monclient(hunting): my global_id is 4192 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.434+0000 7faf8a7fc640 1 -- 172.21.15.7:0/2529181268 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf78003520 con 0x7faf8c142710 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.435+0000 7faf83fff640 10 monclient(hunting): handle_auth_done global_id 4181 payload 274 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.435+0000 7faf83fff640 1 -- 172.21.15.7:0/2529181268 >> v2:172.21.15.7:3300/0 conn(0x7faf8c0a8a60 msgr2=0x7faf8c13f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.435+0000 7faf83fff640 1 --2- 172.21.15.7:0/2529181268 >> v2:172.21.15.7:3300/0 conn(0x7faf8c0a8a60 0x7faf8c13f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.435+0000 7faf83fff640 1 -- 172.21.15.7:0/2529181268 >> v1:172.21.15.177:6789/0 conn(0x7faf8c142710 legacy=0x7faf8c1455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.435+0000 7faf83fff640 10 monclient: _finish_hunting 0 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.435+0000 7faf83fff640 1 monclient: found mon.noname-b 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.435+0000 7faf83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:12.423 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.435+0000 7faf83fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:19:12.424 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.435+0000 7faf83fff640 1 -- 172.21.15.7:0/2529181268 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c13fc40 con 0x7faf8c13f840 2024-01-21T03:19:12.424 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.437+0000 7faf83fff640 1 --2- 172.21.15.7:0/2529181268 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faf8c13f840 0x7faf8c1421d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faf740097a0 tx=0x7faf7402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=391427a4d01da28b server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:12.426 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:09.350 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.428 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf8a7fc640 1 -- 172.21.15.7:0/2529181268 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7402e070 con 0x7faf8c13f840 2024-01-21T03:19:12.428 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:12.428 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf8a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:12.428 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf8a7fc640 10 monclient: dump: 2024-01-21T03:19:12.428 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:12.428 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:12.428 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:12.428 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:12.428 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:12.428 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:12.428 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:12.428 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf8a7fc640 10 monclient: _finish_auth 0 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf8a7fc640 10 monclient: _check_auth_tickets 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf8a7fc640 1 -- 172.21.15.7:0/2529181268 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf740315a0 con 0x7faf8c13f840 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf91289640 5 monclient: authenticate success, global_id 4181 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf91289640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf91289640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf8a7fc640 1 -- 172.21.15.7:0/2529181268 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf74031930 con 0x7faf8c13f840 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf91289640 4 set_mon_vals no callback set 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:12.429 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf8a7fc640 10 monclient: dump: 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf91289640 10 monclient: shutdown 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf91289640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf91289640 1 -- 172.21.15.7:0/2529181268 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faf8c13f840 msgr2=0x7faf8c1421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:12.430 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf91289640 1 --2- 172.21.15.7:0/2529181268 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faf8c13f840 0x7faf8c1421d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faf740097a0 tx=0x7faf7402b840 comp rx=0 tx=0).stop 2024-01-21T03:19:12.431 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf91289640 1 -- 172.21.15.7:0/2529181268 shutdown_connections 2024-01-21T03:19:12.431 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf91289640 1 --2- 172.21.15.7:0/2529181268 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faf8c13f840 0x7faf8c1421d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:12.431 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf91289640 1 --2- 172.21.15.7:0/2529181268 >> v2:172.21.15.7:3300/0 conn(0x7faf8c0a8a60 0x7faf8c13f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:12.431 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.441+0000 7faf91289640 1 -- 172.21.15.7:0/2529181268 >> 172.21.15.7:0/2529181268 conn(0x7faf8c0670b0 msgr2=0x7faf8c0b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:12.431 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.442+0000 7faf91289640 1 -- 172.21.15.7:0/2529181268 shutdown_connections 2024-01-21T03:19:12.431 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.442+0000 7faf91289640 1 -- 172.21.15.7:0/2529181268 wait complete. 2024-01-21T03:19:12.432 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.442+0000 7faf91289640 10 monclient: build_initial_monmap 2024-01-21T03:19:12.432 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.442+0000 7faf91289640 10 monclient: monmap: 2024-01-21T03:19:12.432 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:12.432 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:12.432 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-01-21T03:19:12.432 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-01-21T03:19:12.432 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:12.432 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:12.432 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:12.432 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:12.432 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:12.432 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:12.433 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 1 librados: starting msgr at 2024-01-21T03:19:12.433 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 1 librados: starting objecter 2024-01-21T03:19:12.433 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 1 Processor -- start 2024-01-21T03:19:12.433 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:09.356 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.436 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 1 -- start start 2024-01-21T03:19:12.436 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 1 librados: setting wanted keys 2024-01-21T03:19:12.436 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 1 librados: calling monclient init 2024-01-21T03:19:12.436 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 10 monclient: init 2024-01-21T03:19:12.436 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:12.436 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T03:19:12.436 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf8c13f840 0x7faf8c13e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:12.436 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 10 monclient(hunting): picked mon.noname-b con 0x7faf8c13f840 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:12.436 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf8c142710 0x7faf8c13a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:12.436 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 10 monclient(hunting): picked mon.noname-a con 0x7faf8c142710 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:12.436 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faf8c13c010 0x7faf8c13c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:12.436 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 10 monclient(hunting): picked mon.noname-c con 0x7faf8c13c010 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:12.436 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c13ee60 con 0x7faf8c142710 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c13e9d0 con 0x7faf8c13f840 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c13b350 con 0x7faf8c13c010 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.443+0000 7faf91289640 10 monclient(hunting): authenticate will time out at 1298.759696s 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.444+0000 7faf8affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf8c13f840 0x7faf8c13e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.444+0000 7faf8b7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faf8c13c010 0x7faf8c13c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.444+0000 7faf83fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf8c142710 0x7faf8c13a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.444+0000 7faf8affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf8c13f840 0x7faf8c13e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:53828/0 (socket says 172.21.15.7:53828) 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.444+0000 7faf8affd640 1 -- 172.21.15.7:0/1256544522 learned_addr learned my addr 172.21.15.7:0/1256544522 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.444+0000 7faf8affd640 10 monclient(hunting): get_auth_request con 0x7faf8c13f840 auth_method 0 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.444+0000 7faf8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:12.437 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.444+0000 7faf8affd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.444+0000 7faf8affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.444+0000 7faf8b7fe640 10 monclient(hunting): get_auth_request con 0x7faf8c13c010 auth_method 0 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.444+0000 7faf8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.444+0000 7faf8b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.444+0000 7faf8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.445+0000 7faf83fff640 10 monclient(hunting): get_auth_request con 0x7faf8c142710 auth_method 0 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.445+0000 7faf83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.445+0000 7faf83fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.445+0000 7faf83fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.445+0000 7faf8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.445+0000 7faf8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.445+0000 7faf8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.445+0000 7faf8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:12.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.445+0000 7faf8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.445+0000 7faf8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.445+0000 7faf83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.445+0000 7faf83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.445+0000 7faf83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.447+0000 7faf8affd640 10 monclient(hunting): handle_auth_done global_id 4195 payload 824 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.447+0000 7faf8affd640 1 -- 172.21.15.7:0/1256544522 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faf8c13c010 msgr2=0x7faf8c13c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.447+0000 7faf8affd640 1 --2- 172.21.15.7:0/1256544522 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faf8c13c010 0x7faf8c13c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.447+0000 7faf8affd640 1 -- 172.21.15.7:0/1256544522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf8c142710 msgr2=0x7faf8c13a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.447+0000 7faf8affd640 1 --2- 172.21.15.7:0/1256544522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf8c142710 0x7faf8c13a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.447+0000 7faf8affd640 10 monclient: _finish_hunting 0 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.447+0000 7faf8affd640 1 monclient: found mon.noname-b 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.447+0000 7faf8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.447+0000 7faf8affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:19:12.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.447+0000 7faf8affd640 1 -- 172.21.15.7:0/1256544522 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf74009400 con 0x7faf8c13f840 2024-01-21T03:19:12.441 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.450+0000 7faf8affd640 1 --2- 172.21.15.7:0/1256544522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf8c13f840 0x7faf8c13e1f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faf7c00ea10 tx=0x7faf7c00eed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:12.441 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.450+0000 7faf837fe640 10 client.?.objecter ms_handle_connect 0x7faf8c13f840 2024-01-21T03:19:12.441 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.450+0000 7faf837fe640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:12.441 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf837fe640 1 -- 172.21.15.7:0/1256544522 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7c009670 con 0x7faf8c13f840 2024-01-21T03:19:12.441 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf837fe640 10 monclient: dump: 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf837fe640 10 monclient: _finish_auth 0 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf837fe640 10 monclient: _check_auth_tickets 2024-01-21T03:19:12.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf837fe640 1 -- 172.21.15.7:0/1256544522 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7c00ca00 con 0x7faf8c13f840 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf837fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf837fe640 1 -- 172.21.15.7:0/1256544522 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7c00cdb0 con 0x7faf8c13f840 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf91289640 5 monclient: authenticate success, global_id 4195 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf91289640 10 monclient: _renew_subs 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf91289640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf91289640 1 -- 172.21.15.7:0/1256544522 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8c13b5c0 con 0x7faf8c13f840 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf89ffb640 4 set_mon_vals no callback set 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf837fe640 10 monclient: dump: 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:12.443 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf91289640 10 client.4195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf91289640 10 monclient: _renew_subs 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf91289640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf91289640 1 -- 172.21.15.7:0/1256544522 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8c140700 con 0x7faf8c13f840 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.454+0000 7faf91289640 1 librados: init done 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.455+0000 7faf91289640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:12.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.455+0000 7faf91289640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:12.445 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.455+0000 7faf91289640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:12.445 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.455+0000 7faf91289640 1 -- 172.21.15.7:0/1256544522 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8c1421d0 con 0x7faf8c13f840 2024-01-21T03:19:12.465 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:09.380 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.511 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:09.416 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.520 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:09.447 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.630 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:09.533 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.651 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:09.562 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.668 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:09.593 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.682 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.696+0000 7faf837fe640 1 -- 172.21.15.7:0/1256544522 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) v1 ==== 89142+0+0 (secure 0 0 0) 0x7faf7c013470 con 0x7faf8c13f840 2024-01-21T03:19:12.682 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.696+0000 7faf837fe640 1 --2- 172.21.15.7:0/1256544522 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7faf6c06c7d0 0x7faf6c06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:12.682 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.696+0000 7faf83fff640 1 --2- 172.21.15.7:0/1256544522 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7faf6c06c7d0 0x7faf6c06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:12.683 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.696+0000 7faf83fff640 10 monclient: get_auth_request con 0x7faf6c06c7d0 auth_method 0 2024-01-21T03:19:12.683 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.697+0000 7faf837fe640 10 client.4195.objecter ms_handle_connect 0x7faf6c06c7d0 2024-01-21T03:19:12.683 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.697+0000 7faf83fff640 1 --2- 172.21.15.7:0/1256544522 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7faf6c06c7d0 0x7faf6c06ec90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faf74036000 tx=0x7faf7403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:12.693 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.707+0000 7faf837fe640 1 -- 172.21.15.7:0/1256544522 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7faf7c058c50 con 0x7faf8c13f840 2024-01-21T03:19:12.693 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.707+0000 7faf837fe640 10 client.4195.objecter ms_dispatch 0x7faf8c000b90 osd_map(10..10 src has 1..10) v4 2024-01-21T03:19:12.694 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.707+0000 7faf837fe640 3 client.4195.objecter handle_osd_map got epochs [10,10] > 0 2024-01-21T03:19:12.694 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.707+0000 7faf837fe640 3 client.4195.objecter handle_osd_map decoding full epoch 10 2024-01-21T03:19:12.694 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.707+0000 7faf837fe640 20 client.4195.objecter dump_active .. 0 homeless 2024-01-21T03:19:12.694 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.708+0000 7faf837fe640 1 -- 172.21.15.7:0/1256544522 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7faf7c054f90 con 0x7faf8c13f840 2024-01-21T03:19:12.694 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.708+0000 7faf837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:12.694 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.708+0000 7faf837fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:12.724 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:09.648 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.775 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:09.680 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.796 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:09.724 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.824 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:09.740 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.886 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:09.813 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.894 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:09.797 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:12.901 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.915+0000 7faf91289640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:12.902 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.915+0000 7faf91289640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:12.902 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.915+0000 7faf91289640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:12.902 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.915+0000 7faf91289640 1 -- 172.21.15.7:0/1256544522 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf8c075c30 con 0x7faf8c13f840 2024-01-21T03:19:12.945 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.959+0000 7faf837fe640 1 -- 172.21.15.7:0/1256544522 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7faf7c021090 con 0x7faf8c13f840 2024-01-21T03:19:12.946 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.959+0000 7faf837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:12.946 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.959+0000 7faf837fe640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:12.946 INFO:teuthology.misc.health.smithi007.stdout: 2024-01-21T03:19:12.946 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"8de9e7dd-e982-4048-83dd-ee52350dd56d","created":"2024-01-21T03:18:57.607470+0000","modified":"2024-01-21T03:19:02.952334+0000","last_up_change":"0.000000","last_in_change":"2024-01-21T03:19:02.952334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f4767aa-192e-4ec9-addf-a51c0872da7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"066d7378-561f-4e74-b97b-2ead2fa2625e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f219649b-4d50-40a8-b74e-5a7f16b3c8c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6379a402-760a-4f07-9491-0918752a18fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dc3602e1-b59e-4489-9882-40c914e2a0d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d42afb34-e1ca-4b21-81e9-78d1c8ea454e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1aef2850-8b41-4d39-859e-ed9c4df434b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T03:19:12.948 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 10 librados: watch_flush enter 2024-01-21T03:19:12.948 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 10 librados: watch_flush exit 2024-01-21T03:19:12.949 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 20 client.4195.objecter shutdown clearing up homeless session... 2024-01-21T03:19:12.949 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 10 client.4195.objecter successfully canceled tick 2024-01-21T03:19:12.949 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 1 -- 172.21.15.7:0/1256544522 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7faf6c06c7d0 msgr2=0x7faf6c06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:12.949 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 1 --2- 172.21.15.7:0/1256544522 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7faf6c06c7d0 0x7faf6c06ec90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faf74036000 tx=0x7faf7403d040 comp rx=0 tx=0).stop 2024-01-21T03:19:12.949 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 10 monclient: shutdown 2024-01-21T03:19:12.949 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:12.949 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 1 -- 172.21.15.7:0/1256544522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf8c13f840 msgr2=0x7faf8c13e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:12.949 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 1 --2- 172.21.15.7:0/1256544522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf8c13f840 0x7faf8c13e1f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faf7c00ea10 tx=0x7faf7c00eed0 comp rx=0 tx=0).stop 2024-01-21T03:19:12.949 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 1 -- 172.21.15.7:0/1256544522 shutdown_connections 2024-01-21T03:19:12.949 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 1 --2- 172.21.15.7:0/1256544522 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7faf6c06c7d0 0x7faf6c06ec90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:12.949 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 1 --2- 172.21.15.7:0/1256544522 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faf8c13c010 0x7faf8c13c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:12.950 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 1 --2- 172.21.15.7:0/1256544522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf8c142710 0x7faf8c13a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:12.950 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 1 --2- 172.21.15.7:0/1256544522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf8c13f840 0x7faf8c13e1f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:12.950 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 1 -- 172.21.15.7:0/1256544522 >> 172.21.15.7:0/1256544522 conn(0x7faf8c0a8a60 msgr2=0x7faf8c1390e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:12.950 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 1 -- 172.21.15.7:0/1256544522 shutdown_connections 2024-01-21T03:19:12.950 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 1 -- 172.21.15.7:0/1256544522 wait complete. 2024-01-21T03:19:12.950 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:12.961+0000 7faf91289640 1 librados: shutdown 2024-01-21T03:19:12.957 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-21T03:19:12.963 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:09.887 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.011 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:09.935 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.019 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:09.931 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.056 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:09.961 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.085 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:10.012 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.150 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.065 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.154 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.081 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.159 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:10.062 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.256 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:10.180 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.308 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.220 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.314 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:10.238 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.333 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:10.238 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.374 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:10.301 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.426 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:10.329 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.449 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.364 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.458 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.385 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.544 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:10.468 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.601 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:10.506 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.603 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.515 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.606 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:10.530 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.671 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:10.599 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.690 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:10.593 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.738 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.653 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.762 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.689 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.831 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:10.755 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.868 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:10.773 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.896 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:10.820 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.897 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.809 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.961 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:10.888 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:13.963 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:10.865 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.033 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.948 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.060 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.987 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.122 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:11.046 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.186 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:11.098 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.253 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.180 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.322 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:11.237 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.358 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:11.285 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.362 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:11.266 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.416 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:11.321 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.422 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:11.347 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.451 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:11.353 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.475 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:11.387 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.477 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:11.401 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.532 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:11.435 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.549 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.476 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.614 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:11.529 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.646 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:11.573 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.673 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:11.598 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.746 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:11.670 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:14.764 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:11.676 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:15.076 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:12.003 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:15.135 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:12.063 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:15.164 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:12.079 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:15.222 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:12.137 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:15.236 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:12.164 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:15.319 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:12.246 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:15.414 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:12.326 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:15.471 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:12.383 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:19.552 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:16.455 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:19.645 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:16.550 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:19.692 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:16.616 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:19.706 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:16.631 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:19.763 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T03:19:19.865 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.879+0000 7ffa51646640 10 monclient: get_monmap_and_config 2024-01-21T03:19:19.865 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.879+0000 7ffa51646640 10 monclient: build_initial_monmap 2024-01-21T03:19:19.865 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.879+0000 7ffa51646640 10 monclient: monmap: 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:19:19.880132+0000 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:19:19.880132+0000 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.879+0000 7ffa51646640 1 Processor -- start 2024-01-21T03:19:19.866 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.879+0000 7ffa51646640 1 -- start start 2024-01-21T03:19:19.867 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.879+0000 7ffa51646640 10 monclient: init 2024-01-21T03:19:19.868 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:19.868 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-01-21T03:19:19.868 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa4c0a8a60 addr v1:172.21.15.7:6789/0 2024-01-21T03:19:19.868 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffa4c13fa10 0x7ffa4c1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:19.868 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 10 monclient(hunting): picked mon.noname-c con 0x7ffa4c13fa10 addr v2:172.21.15.177:3300/0 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffa4c142710 0x7ffa4c142b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 10 monclient(hunting): picked mon.noname-a con 0x7ffa4c142710 addr v2:172.21.15.7:3300/0 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c0b1de0 con 0x7ffa4c0a8a60 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4c0a8e40 con 0x7ffa4c142710 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4c0aa480 con 0x7ffa4c13fa10 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.880+0000 7ffa51646640 10 monclient(hunting): authenticate will time out at 1306.196373s 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.881+0000 7ffa4a7fc640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffa4c13fa10 0x7ffa4c1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.881+0000 7ffa4b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffa4c142710 0x7ffa4c142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.881+0000 7ffa4a7fc640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffa4c13fa10 0x7ffa4c1421d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:35180/0 (socket says 172.21.15.7:35180) 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.881+0000 7ffa4a7fc640 1 -- 172.21.15.7:0/312736351 learned_addr learned my addr 172.21.15.7:0/312736351 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:19.869 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.881+0000 7ffa4a7fc640 10 monclient(hunting): get_auth_request con 0x7ffa4c13fa10 auth_method 0 2024-01-21T03:19:19.870 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.881+0000 7ffa4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:19.870 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.881+0000 7ffa4a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:19.870 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.881+0000 7ffa4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:19.870 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.882+0000 7ffa4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:19.870 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.882+0000 7ffa4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:19.870 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.882+0000 7ffa4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:19.870 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.882+0000 7ffa4b7fe640 10 monclient(hunting): get_auth_request con 0x7ffa4c142710 auth_method 0 2024-01-21T03:19:19.870 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.882+0000 7ffa4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:19.870 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.882+0000 7ffa4b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:19.870 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.882+0000 7ffa4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:19.871 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.883+0000 7ffa4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:19.871 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.883+0000 7ffa4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:19.871 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.883+0000 7ffa4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:19.871 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.883+0000 7ffa4a7fc640 10 monclient(hunting): handle_auth_done global_id 4198 payload 274 2024-01-21T03:19:19.871 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.883+0000 7ffa4a7fc640 1 -- 172.21.15.7:0/312736351 >> v2:172.21.15.7:3300/0 conn(0x7ffa4c142710 msgr2=0x7ffa4c142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:19.871 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.883+0000 7ffa4a7fc640 1 --2- 172.21.15.7:0/312736351 >> v2:172.21.15.7:3300/0 conn(0x7ffa4c142710 0x7ffa4c142b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:19.871 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.883+0000 7ffa4a7fc640 1 -- 172.21.15.7:0/312736351 >> v1:172.21.15.7:6789/0 conn(0x7ffa4c0a8a60 legacy=0x7ffa4c13f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:19.871 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.883+0000 7ffa4a7fc640 10 monclient: _finish_hunting 0 2024-01-21T03:19:19.871 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.883+0000 7ffa4a7fc640 1 monclient: found mon.noname-c 2024-01-21T03:19:19.871 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.883+0000 7ffa4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:19.871 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.883+0000 7ffa4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:19:19.871 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.883+0000 7ffa4a7fc640 1 -- 172.21.15.7:0/312736351 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c0aa6f0 con 0x7ffa4c13fa10 2024-01-21T03:19:19.872 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.885+0000 7ffa4a7fc640 1 --2- 172.21.15.7:0/312736351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa4c13fa10 0x7ffa4c1421d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0a9050 tx=0x7ffa4002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16f79fdeda63acac server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:19.878 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa49ffb640 1 -- 172.21.15.7:0/312736351 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa40031870 con 0x7ffa4c13fa10 2024-01-21T03:19:19.878 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:19.878 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa49ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:19:19.878 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa49ffb640 10 monclient: dump: 2024-01-21T03:19:19.878 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:19.878 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:19.878 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:19.878 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:19.878 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:19.878 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa49ffb640 10 monclient: _finish_auth 0 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa49ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa49ffb640 1 -- 172.21.15.7:0/312736351 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa400319d0 con 0x7ffa4c13fa10 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa49ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa49ffb640 1 -- 172.21.15.7:0/312736351 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa40031d60 con 0x7ffa4c13fa10 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa49ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa49ffb640 10 monclient: dump: 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:19.879 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa51646640 5 monclient: authenticate success, global_id 4198 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa51646640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.890+0000 7ffa51646640 4 set_mon_vals no callback set 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.891+0000 7ffa51646640 10 monclient: shutdown 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.891+0000 7ffa51646640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.891+0000 7ffa51646640 1 -- 172.21.15.7:0/312736351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa4c13fa10 msgr2=0x7ffa4c1421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:19.880 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.891+0000 7ffa51646640 1 --2- 172.21.15.7:0/312736351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa4c13fa10 0x7ffa4c1421d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0a9050 tx=0x7ffa4002b910 comp rx=0 tx=0).stop 2024-01-21T03:19:19.881 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.891+0000 7ffa51646640 1 -- 172.21.15.7:0/312736351 shutdown_connections 2024-01-21T03:19:19.881 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.891+0000 7ffa51646640 1 --2- 172.21.15.7:0/312736351 >> v2:172.21.15.7:3300/0 conn(0x7ffa4c142710 0x7ffa4c142b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:19.881 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.891+0000 7ffa51646640 1 --2- 172.21.15.7:0/312736351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa4c13fa10 0x7ffa4c1421d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:19.881 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.891+0000 7ffa51646640 1 -- 172.21.15.7:0/312736351 >> 172.21.15.7:0/312736351 conn(0x7ffa4c0670b0 msgr2=0x7ffa4c0b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:19.881 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.891+0000 7ffa51646640 1 -- 172.21.15.7:0/312736351 shutdown_connections 2024-01-21T03:19:19.881 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.891+0000 7ffa51646640 1 -- 172.21.15.7:0/312736351 wait complete. 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.893+0000 7ffa51646640 10 monclient: build_initial_monmap 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.893+0000 7ffa51646640 10 monclient: monmap: 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.893+0000 7ffa51646640 1 librados: starting msgr at 2024-01-21T03:19:19.887 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.893+0000 7ffa51646640 1 librados: starting objecter 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.894+0000 7ffa51646640 1 Processor -- start 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.894+0000 7ffa51646640 1 -- start start 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.894+0000 7ffa51646640 1 librados: setting wanted keys 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.894+0000 7ffa51646640 1 librados: calling monclient init 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.894+0000 7ffa51646640 10 monclient: init 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c13fa10 0x7ffa4c1460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 10 monclient(hunting): picked mon.noname-a con 0x7ffa4c13fa10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa4c142710 0x7ffa4c146900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 10 monclient(hunting): picked mon.noname-c con 0x7ffa4c142710 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa4c143600 0x7ffa4c143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 10 monclient(hunting): picked mon.noname-b con 0x7ffa4c143600 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:19.888 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c148270 con 0x7ffa4c13fa10 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c147060 con 0x7ffa4c143600 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c143ff0 con 0x7ffa4c142710 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa51646640 10 monclient(hunting): authenticate will time out at 1306.211519s 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa4b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa4c143600 0x7ffa4c143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.895+0000 7ffa4affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c13fa10 0x7ffa4c1460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa4c143600 0x7ffa4c143ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:35190/0 (socket says 172.21.15.7:35190) 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4b7fe640 1 -- 172.21.15.7:0/3181341112 learned_addr learned my addr 172.21.15.7:0/3181341112 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4b7fe640 10 monclient(hunting): get_auth_request con 0x7ffa4c143600 auth_method 0 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:19.889 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4a7fc640 1 --2- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa4c142710 0x7ffa4c146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4affd640 10 monclient(hunting): get_auth_request con 0x7ffa4c13fa10 auth_method 0 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4affd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4a7fc640 10 monclient(hunting): get_auth_request con 0x7ffa4c142710 auth_method 0 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.896+0000 7ffa4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:19.890 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.897+0000 7ffa4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:19.891 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.897+0000 7ffa4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:19.891 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.897+0000 7ffa4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:19.891 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.899+0000 7ffa4affd640 10 monclient(hunting): handle_auth_done global_id 4188 payload 824 2024-01-21T03:19:19.891 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.899+0000 7ffa4affd640 1 -- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa4c142710 msgr2=0x7ffa4c146900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:19.891 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.899+0000 7ffa4affd640 1 --2- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa4c142710 0x7ffa4c146900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:19.891 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.899+0000 7ffa4affd640 1 -- 172.21.15.7:0/3181341112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa4c143600 msgr2=0x7ffa4c143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:19.891 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.899+0000 7ffa4affd640 1 --2- 172.21.15.7:0/3181341112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa4c143600 0x7ffa4c143ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:19.891 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.899+0000 7ffa4affd640 10 monclient: _finish_hunting 0 2024-01-21T03:19:19.891 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.899+0000 7ffa4affd640 1 monclient: found mon.noname-a 2024-01-21T03:19:19.891 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.899+0000 7ffa4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:19.891 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.899+0000 7ffa4affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:19:19.891 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.899+0000 7ffa4affd640 1 -- 172.21.15.7:0/3181341112 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa400094d0 con 0x7ffa4c13fa10 2024-01-21T03:19:19.894 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.902+0000 7ffa4affd640 1 --2- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c13fa10 0x7ffa4c1460e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00c3c0 tx=0x7ffa3c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:19.894 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.902+0000 7ffa337fe640 10 client.?.objecter ms_handle_connect 0x7ffa4c13fa10 2024-01-21T03:19:19.894 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.902+0000 7ffa337fe640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa337fe640 1 -- 172.21.15.7:0/3181341112 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c0052d0 con 0x7ffa4c13fa10 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa337fe640 10 monclient: dump: 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:19.895 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa337fe640 10 monclient: _finish_auth 0 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa337fe640 10 monclient: _check_auth_tickets 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa337fe640 1 -- 172.21.15.7:0/3181341112 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3c005430 con 0x7ffa4c13fa10 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa337fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa337fe640 1 -- 172.21.15.7:0/3181341112 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c003830 con 0x7ffa4c13fa10 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa337fe640 10 monclient: dump: 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:19.896 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.906+0000 7ffa497fa640 4 set_mon_vals no callback set 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.907+0000 7ffa51646640 5 monclient: authenticate success, global_id 4188 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.907+0000 7ffa51646640 10 monclient: _renew_subs 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.907+0000 7ffa51646640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.907+0000 7ffa51646640 1 -- 172.21.15.7:0/3181341112 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c148470 con 0x7ffa4c13fa10 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.907+0000 7ffa51646640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.907+0000 7ffa51646640 10 monclient: _renew_subs 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.907+0000 7ffa51646640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.907+0000 7ffa51646640 1 -- 172.21.15.7:0/3181341112 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c1474f0 con 0x7ffa4c13fa10 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.907+0000 7ffa51646640 1 librados: init done 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.908+0000 7ffa51646640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:19.897 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.908+0000 7ffa51646640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:19.898 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.908+0000 7ffa51646640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:19.898 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.908+0000 7ffa51646640 1 -- 172.21.15.7:0/3181341112 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c0aac90 con 0x7ffa4c13fa10 2024-01-21T03:19:19.918 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.932+0000 7ffa337fe640 1 -- 172.21.15.7:0/3181341112 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) v1 ==== 89142+0+0 (secure 0 0 0) 0x7ffa3c003990 con 0x7ffa4c13fa10 2024-01-21T03:19:19.919 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.933+0000 7ffa337fe640 1 --2- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ffa3806c7d0 0x7ffa3806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:19.919 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.933+0000 7ffa4a7fc640 1 --2- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ffa3806c7d0 0x7ffa3806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:19.919 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.933+0000 7ffa4a7fc640 10 monclient: get_auth_request con 0x7ffa3806c7d0 auth_method 0 2024-01-21T03:19:19.919 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.933+0000 7ffa337fe640 10 client.4188.objecter ms_handle_connect 0x7ffa3806c7d0 2024-01-21T03:19:19.919 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.933+0000 7ffa4a7fc640 1 --2- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ffa3806c7d0 0x7ffa3806ec90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ffa40036000 tx=0x7ffa4003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:19.921 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.935+0000 7ffa337fe640 1 -- 172.21.15.7:0/3181341112 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7ffa3c057900 con 0x7ffa4c13fa10 2024-01-21T03:19:19.921 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.935+0000 7ffa337fe640 10 client.4188.objecter ms_dispatch 0x7ffa4c000b90 osd_map(10..10 src has 1..10) v4 2024-01-21T03:19:19.921 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.935+0000 7ffa337fe640 3 client.4188.objecter handle_osd_map got epochs [10,10] > 0 2024-01-21T03:19:19.921 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.935+0000 7ffa337fe640 3 client.4188.objecter handle_osd_map decoding full epoch 10 2024-01-21T03:19:19.922 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:19.935+0000 7ffa337fe640 20 client.4188.objecter dump_active .. 0 homeless 2024-01-21T03:19:20.175 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.189+0000 7ffa337fe640 1 -- 172.21.15.7:0/3181341112 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ffa3c053c40 con 0x7ffa4c13fa10 2024-01-21T03:19:20.176 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.189+0000 7ffa337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:20.176 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.189+0000 7ffa337fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:20.445 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.459+0000 7ffa51646640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:20.445 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.459+0000 7ffa51646640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:20.445 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.459+0000 7ffa51646640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:20.446 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.459+0000 7ffa51646640 1 -- 172.21.15.7:0/3181341112 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa4c0aac90 con 0x7ffa4c13fa10 2024-01-21T03:19:20.493 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.507+0000 7ffa337fe640 1 -- 172.21.15.7:0/3181341112 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7ffa4c0aac90 con 0x7ffa4c13fa10 2024-01-21T03:19:20.494 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.507+0000 7ffa337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:20.494 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.507+0000 7ffa337fe640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:20.494 INFO:teuthology.misc.health.smithi007.stdout: 2024-01-21T03:19:20.494 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"8de9e7dd-e982-4048-83dd-ee52350dd56d","created":"2024-01-21T03:18:57.607470+0000","modified":"2024-01-21T03:19:02.952334+0000","last_up_change":"0.000000","last_in_change":"2024-01-21T03:19:02.952334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f4767aa-192e-4ec9-addf-a51c0872da7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"066d7378-561f-4e74-b97b-2ead2fa2625e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f219649b-4d50-40a8-b74e-5a7f16b3c8c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6379a402-760a-4f07-9491-0918752a18fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dc3602e1-b59e-4489-9882-40c914e2a0d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d42afb34-e1ca-4b21-81e9-78d1c8ea454e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1aef2850-8b41-4d39-859e-ed9c4df434b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T03:19:20.496 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 10 librados: watch_flush enter 2024-01-21T03:19:20.496 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 10 librados: watch_flush exit 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 20 client.4188.objecter shutdown clearing up homeless session... 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 10 client.4188.objecter successfully canceled tick 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 1 -- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ffa3806c7d0 msgr2=0x7ffa3806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 1 --2- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ffa3806c7d0 0x7ffa3806ec90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ffa40036000 tx=0x7ffa4003c040 comp rx=0 tx=0).stop 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 10 monclient: shutdown 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 1 -- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c13fa10 msgr2=0x7ffa4c1460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 1 --2- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c13fa10 0x7ffa4c1460e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00c3c0 tx=0x7ffa3c00c880 comp rx=0 tx=0).stop 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 1 -- 172.21.15.7:0/3181341112 shutdown_connections 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 1 --2- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ffa3806c7d0 0x7ffa3806ec90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 1 --2- 172.21.15.7:0/3181341112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa4c143600 0x7ffa4c143ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 1 --2- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa4c142710 0x7ffa4c146900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 1 --2- 172.21.15.7:0/3181341112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c13fa10 0x7ffa4c1460e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 1 -- 172.21.15.7:0/3181341112 >> 172.21.15.7:0/3181341112 conn(0x7ffa4c0a8a60 msgr2=0x7ffa4c073df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:20.497 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 1 -- 172.21.15.7:0/3181341112 shutdown_connections 2024-01-21T03:19:20.498 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 1 -- 172.21.15.7:0/3181341112 wait complete. 2024-01-21T03:19:20.498 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:20.510+0000 7ffa51646640 1 librados: shutdown 2024-01-21T03:19:20.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-21T03:19:20.994 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:17.922 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:21.069 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:17.997 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:21.095 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:18.011 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:21.391 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:18.303 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:24.358 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:21.282 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:24.362 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:21.286 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:24.446 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:21.370 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:24.450 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:21.375 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:24.535 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:21.440 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:24.539 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:21.444 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:24.690 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:21.593 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:24.695 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:21.598 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:26.050 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:22.978 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:26.054 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:22.982 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:26.291 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:23.207 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:26.298 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:23.213 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:26.381 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:23.308 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:26.388 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:23.315 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:26.484 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:23.396 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:26.490 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:23.402 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:27.316 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T03:19:27.438 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.452+0000 7f6d42ec9640 10 monclient: get_monmap_and_config 2024-01-21T03:19:27.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.452+0000 7f6d42ec9640 10 monclient: build_initial_monmap 2024-01-21T03:19:27.439 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.452+0000 7f6d42ec9640 10 monclient: monmap: 2024-01-21T03:19:27.439 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:27.439 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:27.439 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:19:27.453584+0000 2024-01-21T03:19:27.439 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:19:27.453584+0000 2024-01-21T03:19:27.439 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:27.439 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:27.439 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:27.439 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:27.439 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:27.439 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:27.439 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:27.440 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:27.441 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.453+0000 7f6d42ec9640 1 Processor -- start 2024-01-21T03:19:27.441 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.453+0000 7f6d42ec9640 1 -- start start 2024-01-21T03:19:27.441 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.453+0000 7f6d42ec9640 10 monclient: init 2024-01-21T03:19:27.441 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:27.441 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8a60 0x7f6d3c13f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f6d3c0a8a60 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d3c13f840 0x7f6d3c1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f6d3c13f840 addr v2:172.21.15.7:3300/0 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d3c142710 addr v1:172.21.15.7:6789/0 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d3c0b1de0 con 0x7f6d3c142710 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3c0a8e40 con 0x7f6d3c13f840 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c0aa480 con 0x7f6d3c0a8a60 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.454+0000 7f6d42ec9640 10 monclient(hunting): authenticate will time out at 1313.770406s 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.455+0000 7f6d40c65640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8a60 0x7f6d3c13f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.455+0000 7f6d3bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d3c13f840 0x7f6d3c1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:27.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.455+0000 7f6d40c65640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8a60 0x7f6d3c13f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:35236/0 (socket says 172.21.15.7:35236) 2024-01-21T03:19:27.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.455+0000 7f6d40c65640 1 -- 172.21.15.7:0/1044935894 learned_addr learned my addr 172.21.15.7:0/1044935894 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:27.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.455+0000 7f6d40c65640 10 monclient(hunting): get_auth_request con 0x7f6d3c0a8a60 auth_method 0 2024-01-21T03:19:27.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.455+0000 7f6d40c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:27.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.455+0000 7f6d40c65640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:27.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.455+0000 7f6d40c65640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:27.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.456+0000 7f6d3bfff640 10 monclient(hunting): get_auth_request con 0x7f6d3c13f840 auth_method 0 2024-01-21T03:19:27.443 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.456+0000 7f6d3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:27.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.456+0000 7f6d3bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:27.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.456+0000 7f6d3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:27.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.456+0000 7f6d40c65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:27.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.456+0000 7f6d40c65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:27.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.456+0000 7f6d40c65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:27.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.456+0000 7f6d3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:27.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.456+0000 7f6d3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:27.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.457+0000 7f6d3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:27.444 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.458+0000 7f6d40c65640 10 monclient(hunting): handle_auth_done global_id 4190 payload 274 2024-01-21T03:19:27.445 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.458+0000 7f6d40c65640 1 -- 172.21.15.7:0/1044935894 >> v2:172.21.15.7:3300/0 conn(0x7f6d3c13f840 msgr2=0x7f6d3c1421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:27.445 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.458+0000 7f6d40c65640 1 --2- 172.21.15.7:0/1044935894 >> v2:172.21.15.7:3300/0 conn(0x7f6d3c13f840 0x7f6d3c1421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:27.445 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.458+0000 7f6d40c65640 1 -- 172.21.15.7:0/1044935894 >> v1:172.21.15.7:6789/0 conn(0x7f6d3c142710 legacy=0x7f6d3c1455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:27.445 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.458+0000 7f6d40c65640 10 monclient: _finish_hunting 0 2024-01-21T03:19:27.445 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.458+0000 7f6d40c65640 1 monclient: found mon.noname-b 2024-01-21T03:19:27.445 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.458+0000 7f6d40c65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:27.445 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.458+0000 7f6d40c65640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:19:27.445 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.458+0000 7f6d40c65640 1 -- 172.21.15.7:0/1044935894 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3c13fc40 con 0x7f6d3c0a8a60 2024-01-21T03:19:27.447 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.461+0000 7f6d40c65640 1 --2- 172.21.15.7:0/1044935894 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8a60 0x7f6d3c13f300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009870 tx=0x7f6d3002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0a2c69efab3db66 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:27.452 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.466+0000 7f6d3b7fe640 1 -- 172.21.15.7:0/1044935894 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d300317c0 con 0x7f6d3c0a8a60 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.466+0000 7f6d3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d3b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d3b7fe640 10 monclient: dump: 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:27.453 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d3b7fe640 10 monclient: _finish_auth 0 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d3b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d3b7fe640 1 -- 172.21.15.7:0/1044935894 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d30031920 con 0x7f6d3c0a8a60 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d42ec9640 5 monclient: authenticate success, global_id 4190 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d42ec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d42ec9640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d3b7fe640 1 -- 172.21.15.7:0/1044935894 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30031cb0 con 0x7f6d3c0a8a60 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d42ec9640 4 set_mon_vals no callback set 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d3b7fe640 10 monclient: dump: 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:27.454 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:27.455 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:27.455 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:27.455 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:27.455 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:27.455 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:27.455 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:27.455 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:27.462 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d42ec9640 10 monclient: shutdown 2024-01-21T03:19:27.462 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d42ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:27.462 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d42ec9640 1 -- 172.21.15.7:0/1044935894 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8a60 msgr2=0x7f6d3c13f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:27.462 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.467+0000 7f6d42ec9640 1 --2- 172.21.15.7:0/1044935894 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8a60 0x7f6d3c13f300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009870 tx=0x7f6d3002b860 comp rx=0 tx=0).stop 2024-01-21T03:19:27.462 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.468+0000 7f6d42ec9640 1 -- 172.21.15.7:0/1044935894 shutdown_connections 2024-01-21T03:19:27.462 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.468+0000 7f6d42ec9640 1 --2- 172.21.15.7:0/1044935894 >> v2:172.21.15.7:3300/0 conn(0x7f6d3c13f840 0x7f6d3c1421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:27.463 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.468+0000 7f6d42ec9640 1 --2- 172.21.15.7:0/1044935894 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c0a8a60 0x7f6d3c13f300 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:27.463 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.468+0000 7f6d42ec9640 1 -- 172.21.15.7:0/1044935894 >> 172.21.15.7:0/1044935894 conn(0x7f6d3c0670b0 msgr2=0x7f6d3c0b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:27.463 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.468+0000 7f6d42ec9640 1 -- 172.21.15.7:0/1044935894 shutdown_connections 2024-01-21T03:19:27.463 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.468+0000 7f6d42ec9640 1 -- 172.21.15.7:0/1044935894 wait complete. 2024-01-21T03:19:27.463 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.469+0000 7f6d42ec9640 10 monclient: build_initial_monmap 2024-01-21T03:19:27.463 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.469+0000 7f6d42ec9640 10 monclient: monmap: 2024-01-21T03:19:27.463 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:27.463 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:27.463 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-01-21T03:19:27.463 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-01-21T03:19:27.463 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:27.463 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:27.463 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:27.464 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:27.464 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:27.464 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:27.464 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.469+0000 7f6d42ec9640 1 librados: starting msgr at 2024-01-21T03:19:27.464 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.469+0000 7f6d42ec9640 1 librados: starting objecter 2024-01-21T03:19:27.464 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.469+0000 7f6d42ec9640 1 Processor -- start 2024-01-21T03:19:27.464 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.469+0000 7f6d42ec9640 1 -- start start 2024-01-21T03:19:27.464 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.469+0000 7f6d42ec9640 1 librados: setting wanted keys 2024-01-21T03:19:27.465 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.469+0000 7f6d42ec9640 1 librados: calling monclient init 2024-01-21T03:19:27.465 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.470+0000 7f6d42ec9640 10 monclient: init 2024-01-21T03:19:27.465 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:27.465 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:19:27.465 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d3c13f840 0x7f6d3c13e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:27.465 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f6d3c13f840 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:27.465 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c142710 0x7f6d3c1daaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:27.465 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f6d3c142710 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:27.466 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d3c1db030 0x7f6d3c1359d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:27.466 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f6d3c1db030 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:27.466 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:27.466 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c1384d0 con 0x7f6d3c1db030 2024-01-21T03:19:27.466 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:27.466 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c138960 con 0x7f6d3c13f840 2024-01-21T03:19:27.466 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:27.466 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c1379f0 con 0x7f6d3c142710 2024-01-21T03:19:27.467 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:27.467 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.471+0000 7f6d42ec9640 10 monclient(hunting): authenticate will time out at 1313.787709s 2024-01-21T03:19:27.467 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d41466640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d3c1db030 0x7f6d3c1359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:27.467 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d3bfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c142710 0x7f6d3c1daaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:27.467 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d40c65640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d3c13f840 0x7f6d3c13e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:27.467 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d40c65640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d3c13f840 0x7f6d3c13e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:42642/0 (socket says 172.21.15.7:42642) 2024-01-21T03:19:27.467 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d40c65640 1 -- 172.21.15.7:0/3121712835 learned_addr learned my addr 172.21.15.7:0/3121712835 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:27.467 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d40c65640 10 monclient(hunting): get_auth_request con 0x7f6d3c13f840 auth_method 0 2024-01-21T03:19:27.468 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d40c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:27.468 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d40c65640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:27.468 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d40c65640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:27.468 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d41466640 10 monclient(hunting): get_auth_request con 0x7f6d3c1db030 auth_method 0 2024-01-21T03:19:27.468 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d41466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:27.468 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d41466640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:27.468 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d41466640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:27.468 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d3bfff640 10 monclient(hunting): get_auth_request con 0x7f6d3c142710 auth_method 0 2024-01-21T03:19:27.469 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:27.469 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d3bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:27.469 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:27.469 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d40c65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:27.469 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d40c65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:27.469 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.472+0000 7f6d40c65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:27.469 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.473+0000 7f6d41466640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:27.469 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.473+0000 7f6d41466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:27.470 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.473+0000 7f6d41466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:27.470 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.473+0000 7f6d3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:27.470 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.473+0000 7f6d3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:27.470 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.473+0000 7f6d3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:27.470 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.475+0000 7f6d40c65640 10 monclient(hunting): handle_auth_done global_id 4204 payload 824 2024-01-21T03:19:27.470 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.475+0000 7f6d40c65640 1 -- 172.21.15.7:0/3121712835 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c142710 msgr2=0x7f6d3c1daaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:27.470 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.475+0000 7f6d40c65640 1 --2- 172.21.15.7:0/3121712835 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c142710 0x7f6d3c1daaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:27.470 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.475+0000 7f6d40c65640 1 -- 172.21.15.7:0/3121712835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d3c1db030 msgr2=0x7f6d3c1359d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:27.470 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.475+0000 7f6d40c65640 1 --2- 172.21.15.7:0/3121712835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d3c1db030 0x7f6d3c1359d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:27.470 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.475+0000 7f6d40c65640 10 monclient: _finish_hunting 0 2024-01-21T03:19:27.470 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.475+0000 7f6d40c65640 1 monclient: found mon.noname-b 2024-01-21T03:19:27.470 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.475+0000 7f6d40c65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:27.470 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.475+0000 7f6d40c65640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:19:27.471 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.475+0000 7f6d40c65640 1 -- 172.21.15.7:0/3121712835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d300094d0 con 0x7f6d3c13f840 2024-01-21T03:19:27.474 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.478+0000 7f6d40c65640 1 --2- 172.21.15.7:0/3121712835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d3c13f840 0x7f6d3c13e720 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6d30000c00 tx=0x7f6d300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:27.474 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.478+0000 7f6d397fa640 10 client.?.objecter ms_handle_connect 0x7f6d3c13f840 2024-01-21T03:19:27.474 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.478+0000 7f6d397fa640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:27.474 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d397fa640 1 -- 172.21.15.7:0/3121712835 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30032860 con 0x7f6d3c13f840 2024-01-21T03:19:27.474 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:27.474 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d397fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:27.474 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d397fa640 10 monclient: dump: 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d397fa640 10 monclient: _finish_auth 0 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d397fa640 10 monclient: _check_auth_tickets 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d397fa640 1 -- 172.21.15.7:0/3121712835 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d300329e0 con 0x7f6d3c13f840 2024-01-21T03:19:27.475 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d397fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d397fa640 1 -- 172.21.15.7:0/3121712835 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30032d90 con 0x7f6d3c13f840 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d397fa640 10 monclient: dump: 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:27.476 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d42ec9640 5 monclient: authenticate success, global_id 4204 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d3affd640 4 set_mon_vals no callback set 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d42ec9640 10 monclient: _renew_subs 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d42ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.482+0000 7f6d42ec9640 1 -- 172.21.15.7:0/3121712835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d3c1db4e0 con 0x7f6d3c13f840 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.483+0000 7f6d42ec9640 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.483+0000 7f6d42ec9640 10 monclient: _renew_subs 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.483+0000 7f6d42ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.483+0000 7f6d42ec9640 1 -- 172.21.15.7:0/3121712835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3c136ad0 con 0x7f6d3c13f840 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.483+0000 7f6d42ec9640 1 librados: init done 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.484+0000 7f6d42ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.484+0000 7f6d42ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.484+0000 7f6d42ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:27.477 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.484+0000 7f6d42ec9640 1 -- 172.21.15.7:0/3121712835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d3c1382d0 con 0x7f6d3c13f840 2024-01-21T03:19:27.492 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.506+0000 7f6d397fa640 1 -- 172.21.15.7:0/3121712835 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) v1 ==== 89142+0+0 (secure 0 0 0) 0x7f6d3004e450 con 0x7f6d3c13f840 2024-01-21T03:19:27.494 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.507+0000 7f6d397fa640 1 --2- 172.21.15.7:0/3121712835 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6d2006c820 0x7f6d2006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:27.494 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.507+0000 7f6d3bfff640 1 --2- 172.21.15.7:0/3121712835 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6d2006c820 0x7f6d2006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:27.494 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.507+0000 7f6d3bfff640 10 monclient: get_auth_request con 0x7f6d2006c820 auth_method 0 2024-01-21T03:19:27.494 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.508+0000 7f6d397fa640 10 client.4204.objecter ms_handle_connect 0x7f6d2006c820 2024-01-21T03:19:27.494 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.508+0000 7f6d3bfff640 1 --2- 172.21.15.7:0/3121712835 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6d2006c820 0x7f6d2006ece0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0b3b80 tx=0x7f6d28009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:27.495 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.509+0000 7f6d397fa640 1 -- 172.21.15.7:0/3121712835 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f6d3007a610 con 0x7f6d3c13f840 2024-01-21T03:19:27.495 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.509+0000 7f6d397fa640 10 client.4204.objecter ms_dispatch 0x7f6d3c000b90 osd_map(10..10 src has 1..10) v4 2024-01-21T03:19:27.495 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.509+0000 7f6d397fa640 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2024-01-21T03:19:27.495 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.509+0000 7f6d397fa640 3 client.4204.objecter handle_osd_map decoding full epoch 10 2024-01-21T03:19:27.495 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.509+0000 7f6d397fa640 20 client.4204.objecter dump_active .. 0 homeless 2024-01-21T03:19:27.914 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.928+0000 7f6d397fa640 1 -- 172.21.15.7:0/3121712835 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6d30076950 con 0x7f6d3c13f840 2024-01-21T03:19:27.914 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.928+0000 7f6d397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:27.914 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:27.928+0000 7f6d397fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:28.223 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.237+0000 7f6d42ec9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:28.224 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.237+0000 7f6d42ec9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:28.224 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.237+0000 7f6d42ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:28.224 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.237+0000 7f6d42ec9640 1 -- 172.21.15.7:0/3121712835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d3c137b30 con 0x7f6d3c13f840 2024-01-21T03:19:28.261 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.275+0000 7f6d397fa640 1 -- 172.21.15.7:0/3121712835 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f6d3c137b30 con 0x7f6d3c13f840 2024-01-21T03:19:28.261 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.275+0000 7f6d397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:28.261 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.275+0000 7f6d397fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:28.261 INFO:teuthology.misc.health.smithi007.stdout: 2024-01-21T03:19:28.261 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"8de9e7dd-e982-4048-83dd-ee52350dd56d","created":"2024-01-21T03:18:57.607470+0000","modified":"2024-01-21T03:19:02.952334+0000","last_up_change":"0.000000","last_in_change":"2024-01-21T03:19:02.952334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f4767aa-192e-4ec9-addf-a51c0872da7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"066d7378-561f-4e74-b97b-2ead2fa2625e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f219649b-4d50-40a8-b74e-5a7f16b3c8c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6379a402-760a-4f07-9491-0918752a18fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dc3602e1-b59e-4489-9882-40c914e2a0d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d42afb34-e1ca-4b21-81e9-78d1c8ea454e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1aef2850-8b41-4d39-859e-ed9c4df434b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T03:19:28.265 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 10 librados: watch_flush enter 2024-01-21T03:19:28.265 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 10 librados: watch_flush exit 2024-01-21T03:19:28.265 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 20 client.4204.objecter shutdown clearing up homeless session... 2024-01-21T03:19:28.265 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 10 client.4204.objecter successfully canceled tick 2024-01-21T03:19:28.265 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 1 -- 172.21.15.7:0/3121712835 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6d2006c820 msgr2=0x7f6d2006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:28.265 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 1 --2- 172.21.15.7:0/3121712835 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6d2006c820 0x7f6d2006ece0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0b3b80 tx=0x7f6d28009190 comp rx=0 tx=0).stop 2024-01-21T03:19:28.265 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 10 monclient: shutdown 2024-01-21T03:19:28.265 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:28.265 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 1 -- 172.21.15.7:0/3121712835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d3c13f840 msgr2=0x7f6d3c13e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:28.265 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 1 --2- 172.21.15.7:0/3121712835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d3c13f840 0x7f6d3c13e720 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6d30000c00 tx=0x7f6d300046c0 comp rx=0 tx=0).stop 2024-01-21T03:19:28.265 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 1 -- 172.21.15.7:0/3121712835 shutdown_connections 2024-01-21T03:19:28.266 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 1 --2- 172.21.15.7:0/3121712835 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6d2006c820 0x7f6d2006ece0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:28.266 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 1 --2- 172.21.15.7:0/3121712835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d3c1db030 0x7f6d3c1359d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:28.266 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 1 --2- 172.21.15.7:0/3121712835 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d3c142710 0x7f6d3c1daaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:28.266 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 1 --2- 172.21.15.7:0/3121712835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d3c13f840 0x7f6d3c13e720 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6d30000c00 tx=0x7f6d300046c0 comp rx=0 tx=0).stop 2024-01-21T03:19:28.266 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.277+0000 7f6d42ec9640 1 -- 172.21.15.7:0/3121712835 >> 172.21.15.7:0/3121712835 conn(0x7f6d3c0a8a60 msgr2=0x7f6d3c073df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:28.266 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.278+0000 7f6d42ec9640 1 -- 172.21.15.7:0/3121712835 shutdown_connections 2024-01-21T03:19:28.266 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.278+0000 7f6d42ec9640 1 -- 172.21.15.7:0/3121712835 wait complete. 2024-01-21T03:19:28.266 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:28.278+0000 7f6d42ec9640 1 librados: shutdown 2024-01-21T03:19:28.281 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-21T03:19:28.436 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:25.360 34522== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:28.694 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:25.618 34523== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:28.765 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:25.670 34524== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:28.932 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:25.834 34525== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:30.329 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:27.256 34485== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:30.426 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:27.342 34487== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:30.698 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:27.626 34486== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:30.958 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:27.870 34488== Warning: unimplemented fcntl command: 1038 2024-01-21T03:19:35.087 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T03:19:35.177 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.191+0000 7f61a37d7640 10 monclient: get_monmap_and_config 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.191+0000 7f61a37d7640 10 monclient: build_initial_monmap 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.191+0000 7f61a37d7640 10 monclient: monmap: 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:19:35.192570+0000 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:19:35.192570+0000 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:35.178 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:35.179 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.192+0000 7f61a37d7640 1 Processor -- start 2024-01-21T03:19:35.180 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.192+0000 7f61a37d7640 1 -- start start 2024-01-21T03:19:35.180 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.192+0000 7f61a37d7640 10 monclient: init 2024-01-21T03:19:35.180 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a37d7640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:35.180 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a37d7640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-01-21T03:19:35.180 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a37d7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f619c0a8a60 0x7f619c13f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:35.180 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a37d7640 10 monclient(hunting): picked mon.noname-c con 0x7f619c0a8a60 addr v2:172.21.15.177:3300/0 2024-01-21T03:19:35.180 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a37d7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f619c13f840 addr v1:172.21.15.177:6789/0 2024-01-21T03:19:35.180 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a37d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f619c1428e0 addr v1:172.21.15.7:6789/0 2024-01-21T03:19:35.180 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a37d7640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c0b1de0 con 0x7f619c1428e0 2024-01-21T03:19:35.181 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a37d7640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c13fc70 con 0x7f619c13f840 2024-01-21T03:19:35.181 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a37d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:35.181 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a37d7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619c0a8e40 con 0x7f619c0a8a60 2024-01-21T03:19:35.181 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a37d7640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:35.181 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a37d7640 10 monclient(hunting): authenticate will time out at 1321.509146s 2024-01-21T03:19:35.181 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a1d74640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f619c1428e0 0x7f619c14c930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49608/0 (socket says 172.21.15.7:49608) 2024-01-21T03:19:35.181 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.193+0000 7f61a1d74640 1 -- 172.21.15.7:0/3882070888 learned_addr learned my addr 172.21.15.7:0/3882070888 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:19:35.181 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.194+0000 7f61a1573640 1 --2- 172.21.15.7:0/3882070888 >> v2:172.21.15.177:3300/0 conn(0x7f619c0a8a60 0x7f619c13f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:35.182 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.194+0000 7f61a1573640 10 monclient(hunting): get_auth_request con 0x7f619c0a8a60 auth_method 0 2024-01-21T03:19:35.182 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.194+0000 7f61a1573640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:35.182 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.194+0000 7f61a1573640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:35.182 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.194+0000 7f61a1573640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:35.182 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.195+0000 7f61a1573640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:35.182 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.195+0000 7f61a1573640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:35.182 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.195+0000 7f61a1573640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:35.183 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.197+0000 7f6193fff640 1 -- 172.21.15.7:0/3882070888 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306029439 0 0) 0x7f619c0b1de0 con 0x7f619c1428e0 2024-01-21T03:19:35.183 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.197+0000 7f6193fff640 10 monclient(hunting): _negotiate 2024-01-21T03:19:35.183 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.197+0000 7f6193fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:35.183 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.197+0000 7f6193fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:35.183 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.197+0000 7f6193fff640 10 monclient(hunting): my global_id is 4197 2024-01-21T03:19:35.183 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.197+0000 7f6193fff640 1 -- 172.21.15.7:0/3882070888 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6188003520 con 0x7f619c1428e0 2024-01-21T03:19:35.186 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.200+0000 7f61a1573640 10 monclient(hunting): handle_auth_done global_id 4207 payload 274 2024-01-21T03:19:35.186 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.200+0000 7f61a1573640 1 -- 172.21.15.7:0/3882070888 >> v1:172.21.15.177:6789/0 conn(0x7f619c13f840 legacy=0x7f619c1421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:35.186 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.200+0000 7f61a1573640 1 -- 172.21.15.7:0/3882070888 >> v1:172.21.15.7:6789/0 conn(0x7f619c1428e0 legacy=0x7f619c14c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:35.186 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.200+0000 7f61a1573640 10 monclient: _finish_hunting 0 2024-01-21T03:19:35.186 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.200+0000 7f61a1573640 1 monclient: found mon.noname-c 2024-01-21T03:19:35.186 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.200+0000 7f61a1573640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:35.186 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.200+0000 7f61a1573640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:19:35.186 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.200+0000 7f61a1573640 1 -- 172.21.15.7:0/3882070888 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c0aa480 con 0x7f619c0a8a60 2024-01-21T03:19:35.190 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.204+0000 7f61a1573640 1 --2- 172.21.15.7:0/3882070888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f619c0a8a60 0x7f619c13f300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f61980097a0 tx=0x7f619802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9f7a145591dea20 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:35.194 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f6193fff640 1 -- 172.21.15.7:0/3882070888 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6198031830 con 0x7f619c0a8a60 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f6193fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f6193fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f6193fff640 10 monclient: dump: 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f6193fff640 10 monclient: _finish_auth 0 2024-01-21T03:19:35.195 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f6193fff640 10 monclient: _check_auth_tickets 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f6193fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f6193fff640 1 -- 172.21.15.7:0/3882070888 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6198031990 con 0x7f619c0a8a60 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f6193fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f6193fff640 1 -- 172.21.15.7:0/3882070888 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6198031d20 con 0x7f619c0a8a60 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f6193fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f6193fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f6193fff640 10 monclient: dump: 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:35.196 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:35.197 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:35.197 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:35.197 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:35.197 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f61a37d7640 5 monclient: authenticate success, global_id 4207 2024-01-21T03:19:35.197 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f61a37d7640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:35.197 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.208+0000 7f61a37d7640 4 set_mon_vals no callback set 2024-01-21T03:19:35.202 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.209+0000 7f61a37d7640 10 monclient: shutdown 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.209+0000 7f61a37d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.209+0000 7f61a37d7640 1 -- 172.21.15.7:0/3882070888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f619c0a8a60 msgr2=0x7f619c13f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.209+0000 7f61a37d7640 1 --2- 172.21.15.7:0/3882070888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f619c0a8a60 0x7f619c13f300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f61980097a0 tx=0x7f619802b8d0 comp rx=0 tx=0).stop 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.209+0000 7f61a37d7640 1 -- 172.21.15.7:0/3882070888 shutdown_connections 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.209+0000 7f61a37d7640 1 --2- 172.21.15.7:0/3882070888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f619c0a8a60 0x7f619c13f300 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.209+0000 7f61a37d7640 1 -- 172.21.15.7:0/3882070888 >> 172.21.15.7:0/3882070888 conn(0x7f619c0670b0 msgr2=0x7f619c0b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.209+0000 7f61a37d7640 1 -- 172.21.15.7:0/3882070888 shutdown_connections 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.209+0000 7f61a37d7640 1 -- 172.21.15.7:0/3882070888 wait complete. 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.210+0000 7f61a37d7640 10 monclient: build_initial_monmap 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.210+0000 7f61a37d7640 10 monclient: monmap: 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-01-21T03:19:35.203 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.210+0000 7f61a37d7640 1 librados: starting msgr at 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.210+0000 7f61a37d7640 1 librados: starting objecter 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.210+0000 7f61a37d7640 1 Processor -- start 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.210+0000 7f61a37d7640 1 -- start start 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.210+0000 7f61a37d7640 1 librados: setting wanted keys 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.210+0000 7f61a37d7640 1 librados: calling monclient init 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.210+0000 7f61a37d7640 10 monclient: init 2024-01-21T03:19:35.204 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f619c13f840 0x7f619c13bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 10 monclient(hunting): picked mon.noname-b con 0x7f619c13f840 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f619c1428e0 0x7f619c1db9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 10 monclient(hunting): picked mon.noname-c con 0x7f619c1428e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f619c1daf50 0x7f619c1d7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 10 monclient(hunting): picked mon.noname-a con 0x7f619c1daf50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c0ac2e0 con 0x7f619c1daf50 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c1db4c0 con 0x7f619c13f840 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c1d7f90 con 0x7f619c1428e0 2024-01-21T03:19:35.205 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.211+0000 7f61a37d7640 10 monclient(hunting): authenticate will time out at 1321.527226s 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a1d74640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f619c1daf50 0x7f619c1d7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a1573640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f619c13f840 0x7f619c13bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a0d72640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f619c1428e0 0x7f619c1db9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a1573640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f619c13f840 0x7f619c13bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:42656/0 (socket says 172.21.15.7:42656) 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a1573640 1 -- 172.21.15.7:0/2338131795 learned_addr learned my addr 172.21.15.7:0/2338131795 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a1573640 10 monclient(hunting): get_auth_request con 0x7f619c13f840 auth_method 0 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a1573640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a1573640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a1573640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a1d74640 10 monclient(hunting): get_auth_request con 0x7f619c1daf50 auth_method 0 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a1d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a1d74640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a1d74640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a0d72640 10 monclient(hunting): get_auth_request con 0x7f619c1428e0 auth_method 0 2024-01-21T03:19:35.206 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a0d72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a0d72640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.212+0000 7f61a0d72640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.213+0000 7f61a1573640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.213+0000 7f61a1573640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.213+0000 7f61a1573640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.213+0000 7f61a1d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.213+0000 7f61a1d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.213+0000 7f61a1d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.213+0000 7f61a0d72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.213+0000 7f61a0d72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.213+0000 7f61a0d72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.215+0000 7f61a1d74640 10 monclient(hunting): handle_auth_done global_id 4200 payload 824 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.215+0000 7f61a1d74640 1 -- 172.21.15.7:0/2338131795 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f619c1428e0 msgr2=0x7f619c1db9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.216+0000 7f61a1d74640 1 --2- 172.21.15.7:0/2338131795 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f619c1428e0 0x7f619c1db9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:35.207 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.216+0000 7f61a1d74640 1 -- 172.21.15.7:0/2338131795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f619c13f840 msgr2=0x7f619c13bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:35.208 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.216+0000 7f61a1d74640 1 --2- 172.21.15.7:0/2338131795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f619c13f840 0x7f619c13bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:35.208 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.216+0000 7f61a1d74640 10 monclient: _finish_hunting 0 2024-01-21T03:19:35.208 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.216+0000 7f61a1d74640 1 monclient: found mon.noname-a 2024-01-21T03:19:35.208 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.216+0000 7f61a1d74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:35.208 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.216+0000 7f61a1d74640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:19:35.208 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.216+0000 7f61a1d74640 1 -- 172.21.15.7:0/2338131795 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6198009400 con 0x7f619c1daf50 2024-01-21T03:19:35.211 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.218+0000 7f61a1d74640 1 --2- 172.21.15.7:0/2338131795 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f619c1daf50 0x7f619c1d7a50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f619400ca70 tx=0x7f619400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:35.211 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.218+0000 7f6191ffb640 10 client.?.objecter ms_handle_connect 0x7f619c1daf50 2024-01-21T03:19:35.211 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.218+0000 7f6191ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:35.211 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f6191ffb640 1 -- 172.21.15.7:0/2338131795 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619400ab70 con 0x7f619c1daf50 2024-01-21T03:19:35.211 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f6191ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:35.211 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f6191ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:19:35.211 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f6191ffb640 10 monclient: dump: 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f6191ffb640 10 monclient: _finish_auth 0 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f6191ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f6191ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f6191ffb640 1 -- 172.21.15.7:0/2338131795 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619400acd0 con 0x7f619c1daf50 2024-01-21T03:19:35.212 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f6191ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f6191ffb640 1 -- 172.21.15.7:0/2338131795 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194012620 con 0x7f619c1daf50 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f6191ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f6191ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f6191ffb640 10 monclient: dump: 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:35.213 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f61937fe640 4 set_mon_vals no callback set 2024-01-21T03:19:35.214 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f61a37d7640 5 monclient: authenticate success, global_id 4200 2024-01-21T03:19:35.214 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f61a37d7640 10 monclient: _renew_subs 2024-01-21T03:19:35.214 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f61a37d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:35.214 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.222+0000 7f61a37d7640 1 -- 172.21.15.7:0/2338131795 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619c0b0b70 con 0x7f619c1daf50 2024-01-21T03:19:35.214 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.223+0000 7f61a37d7640 10 client.4200.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:35.214 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.223+0000 7f61a37d7640 10 monclient: _renew_subs 2024-01-21T03:19:35.214 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.223+0000 7f61a37d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:35.214 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.223+0000 7f61a37d7640 1 -- 172.21.15.7:0/2338131795 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619c136a20 con 0x7f619c1daf50 2024-01-21T03:19:35.214 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.223+0000 7f61a37d7640 1 librados: init done 2024-01-21T03:19:35.214 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.224+0000 7f61a37d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:35.214 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.224+0000 7f61a37d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:35.214 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.224+0000 7f61a37d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:35.214 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.224+0000 7f61a37d7640 1 -- 172.21.15.7:0/2338131795 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619c075c30 con 0x7f619c1daf50 2024-01-21T03:19:35.229 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.243+0000 7f6191ffb640 1 -- 172.21.15.7:0/2338131795 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) v1 ==== 89142+0+0 (secure 0 0 0) 0x7f6194007070 con 0x7f619c1daf50 2024-01-21T03:19:35.229 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.244+0000 7f6191ffb640 1 --2- 172.21.15.7:0/2338131795 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f618006c800 0x7f618006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:35.230 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.244+0000 7f61a1573640 1 --2- 172.21.15.7:0/2338131795 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f618006c800 0x7f618006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:35.230 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.244+0000 7f61a1573640 10 monclient: get_auth_request con 0x7f618006c800 auth_method 0 2024-01-21T03:19:35.230 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.244+0000 7f6191ffb640 10 client.4200.objecter ms_handle_connect 0x7f618006c800 2024-01-21T03:19:35.230 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.244+0000 7f61a1573640 1 --2- 172.21.15.7:0/2338131795 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f618006c800 0x7f618006ecc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f619802b910 tx=0x7f619803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:35.507 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.521+0000 7f6191ffb640 1 -- 172.21.15.7:0/2338131795 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f61940584e0 con 0x7f619c1daf50 2024-01-21T03:19:35.507 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.521+0000 7f6191ffb640 10 client.4200.objecter ms_dispatch 0x7f619c000b90 osd_map(10..10 src has 1..10) v4 2024-01-21T03:19:35.507 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.521+0000 7f6191ffb640 3 client.4200.objecter handle_osd_map got epochs [10,10] > 0 2024-01-21T03:19:35.507 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.521+0000 7f6191ffb640 3 client.4200.objecter handle_osd_map decoding full epoch 10 2024-01-21T03:19:35.507 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.521+0000 7f6191ffb640 20 client.4200.objecter dump_active .. 0 homeless 2024-01-21T03:19:35.508 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.521+0000 7f6191ffb640 1 -- 172.21.15.7:0/2338131795 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6194054820 con 0x7f619c1daf50 2024-01-21T03:19:35.508 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.521+0000 7f6191ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:35.508 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.521+0000 7f6191ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:35.712 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.725+0000 7f61a37d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:35.712 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.725+0000 7f61a37d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:35.712 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.725+0000 7f61a37d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:35.712 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.725+0000 7f61a37d7640 1 -- 172.21.15.7:0/2338131795 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f619c0aa720 con 0x7f619c1daf50 2024-01-21T03:19:35.728 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.742+0000 7f6191ffb640 1 -- 172.21.15.7:0/2338131795 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f6194017030 con 0x7f619c1daf50 2024-01-21T03:19:35.728 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.742+0000 7f6191ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:35.728 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.742+0000 7f6191ffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:35.729 INFO:teuthology.misc.health.smithi007.stdout: 2024-01-21T03:19:35.729 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"8de9e7dd-e982-4048-83dd-ee52350dd56d","created":"2024-01-21T03:18:57.607470+0000","modified":"2024-01-21T03:19:02.952334+0000","last_up_change":"0.000000","last_in_change":"2024-01-21T03:19:02.952334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f4767aa-192e-4ec9-addf-a51c0872da7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"066d7378-561f-4e74-b97b-2ead2fa2625e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f219649b-4d50-40a8-b74e-5a7f16b3c8c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6379a402-760a-4f07-9491-0918752a18fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dc3602e1-b59e-4489-9882-40c914e2a0d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d42afb34-e1ca-4b21-81e9-78d1c8ea454e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1aef2850-8b41-4d39-859e-ed9c4df434b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T03:19:35.730 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.744+0000 7f61a37d7640 10 librados: watch_flush enter 2024-01-21T03:19:35.731 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.744+0000 7f61a37d7640 10 librados: watch_flush exit 2024-01-21T03:19:35.731 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.744+0000 7f61a37d7640 20 client.4200.objecter shutdown clearing up homeless session... 2024-01-21T03:19:35.731 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.744+0000 7f61a37d7640 10 client.4200.objecter successfully canceled tick 2024-01-21T03:19:35.731 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 1 -- 172.21.15.7:0/2338131795 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f618006c800 msgr2=0x7f618006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:35.731 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 1 --2- 172.21.15.7:0/2338131795 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f618006c800 0x7f618006ecc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f619802b910 tx=0x7f619803c040 comp rx=0 tx=0).stop 2024-01-21T03:19:35.731 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 10 monclient: shutdown 2024-01-21T03:19:35.731 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:35.731 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 1 -- 172.21.15.7:0/2338131795 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f619c1daf50 msgr2=0x7f619c1d7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:35.731 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 1 --2- 172.21.15.7:0/2338131795 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f619c1daf50 0x7f619c1d7a50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f619400ca70 tx=0x7f619400cf30 comp rx=0 tx=0).stop 2024-01-21T03:19:35.731 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 1 -- 172.21.15.7:0/2338131795 shutdown_connections 2024-01-21T03:19:35.731 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 1 --2- 172.21.15.7:0/2338131795 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f618006c800 0x7f618006ecc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:35.731 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 1 --2- 172.21.15.7:0/2338131795 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f619c1daf50 0x7f619c1d7a50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:35.731 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 1 --2- 172.21.15.7:0/2338131795 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f619c1428e0 0x7f619c1db9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:35.732 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 1 --2- 172.21.15.7:0/2338131795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f619c13f840 0x7f619c13bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:35.732 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 1 -- 172.21.15.7:0/2338131795 >> 172.21.15.7:0/2338131795 conn(0x7f619c0a8a60 msgr2=0x7f619c139180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:35.732 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 1 -- 172.21.15.7:0/2338131795 shutdown_connections 2024-01-21T03:19:35.732 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 1 -- 172.21.15.7:0/2338131795 wait complete. 2024-01-21T03:19:35.732 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:35.745+0000 7f61a37d7640 1 librados: shutdown 2024-01-21T03:19:35.742 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-21T03:19:36.716 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:19:36.726+0000 5dc2300 -1 osd.4 0 log_to_monitors true 2024-01-21T03:19:36.734 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:36.745+0000 5dc2300 -1 osd.5 0 log_to_monitors true 2024-01-21T03:19:36.787 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:36.797+0000 5dc2300 -1 osd.6 0 log_to_monitors true 2024-01-21T03:19:37.109 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-01-21T03:19:37.120+0000 5dc2300 -1 osd.7 0 log_to_monitors true 2024-01-21T03:19:38.730 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:19:38.740+0000 5dc2300 -1 osd.0 0 log_to_monitors true 2024-01-21T03:19:39.115 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:39.126+0000 5dc2300 -1 osd.1 0 log_to_monitors true 2024-01-21T03:19:39.732 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:19:39.688+0000 5dc2300 -1 osd.2 0 log_to_monitors true 2024-01-21T03:19:39.769 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:19:39.779+0000 5dc2300 -1 osd.3 0 log_to_monitors true 2024-01-21T03:19:40.314 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:19:40.324+0000 c493640 -1 osd.4 0 waiting for initial osdmap 2024-01-21T03:19:40.345 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-01-21T03:19:40.355+0000 ac7d640 -1 osd.7 0 waiting for initial osdmap 2024-01-21T03:19:40.782 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:19:40.782+0000 10252640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T03:19:40.803 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-01-21T03:19:40.804+0000 10252640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T03:19:42.549 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T03:19:42.638 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.652+0000 7f4bc1741640 10 monclient: get_monmap_and_config 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.652+0000 7f4bc1741640 10 monclient: build_initial_monmap 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.652+0000 7f4bc1741640 10 monclient: monmap: 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:19:42.653367+0000 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:19:42.653367+0000 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:42.639 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:42.640 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.653+0000 7f4bc1741640 1 Processor -- start 2024-01-21T03:19:42.640 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.653+0000 7f4bc1741640 1 -- start start 2024-01-21T03:19:42.640 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.653+0000 7f4bc1741640 10 monclient: init 2024-01-21T03:19:42.641 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.654+0000 7f4bc1741640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:42.641 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.654+0000 7f4bc1741640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-01-21T03:19:42.641 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.654+0000 7f4bc1741640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bbc0a8a60 addr v1:172.21.15.7:6789/0 2024-01-21T03:19:42.641 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.654+0000 7f4bc1741640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4bbc13fa10 0x7f4bbc1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:42.641 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.654+0000 7f4bc1741640 10 monclient(hunting): picked mon.noname-a con 0x7f4bbc13fa10 addr v2:172.21.15.7:3300/0 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.654+0000 7f4bc1741640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4bbc142710 addr v1:172.21.15.177:6789/0 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.654+0000 7f4bc1741640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc0b1de0 con 0x7f4bbc0a8a60 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.654+0000 7f4bc1741640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc0aa480 con 0x7f4bbc142710 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.654+0000 7f4bc1741640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.654+0000 7f4bc1741640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bbc0a8e40 con 0x7f4bbc13fa10 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.654+0000 7f4bc1741640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.654+0000 7f4bc1741640 10 monclient(hunting): authenticate will time out at 1328.969928s 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.654+0000 7f4bba7fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4bbc13fa10 0x7f4bbc1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.655+0000 7f4bbb7fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f4bbc142710 0x7f4bbc14c930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:56912/0 (socket says 172.21.15.7:56912) 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.655+0000 7f4bbb7fe640 1 -- 172.21.15.7:0/3070228206 learned_addr learned my addr 172.21.15.7:0/3070228206 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.655+0000 7f4bba7fc640 10 monclient(hunting): get_auth_request con 0x7f4bbc13fa10 auth_method 0 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.655+0000 7f4bba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.655+0000 7f4bba7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:42.642 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.655+0000 7f4bba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:42.643 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.656+0000 7f4bba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:42.643 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.656+0000 7f4bba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:42.643 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.656+0000 7f4bba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:42.644 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.657+0000 7f4bb9ffb640 1 -- 172.21.15.7:0/3070228206 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940805537 0 0) 0x7f4bbc0aa480 con 0x7f4bbc142710 2024-01-21T03:19:42.644 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.657+0000 7f4bb9ffb640 10 monclient(hunting): _negotiate 2024-01-21T03:19:42.644 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.657+0000 7f4bb9ffb640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:42.644 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.657+0000 7f4bb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:42.644 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.657+0000 7f4bb9ffb640 10 monclient(hunting): my global_id is 4234 2024-01-21T03:19:42.644 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.657+0000 7f4bb9ffb640 1 -- 172.21.15.7:0/3070228206 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba4003520 con 0x7f4bbc142710 2024-01-21T03:19:42.646 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.658+0000 7f4bba7fc640 10 monclient(hunting): handle_auth_done global_id 4221 payload 274 2024-01-21T03:19:42.646 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.658+0000 7f4bba7fc640 1 -- 172.21.15.7:0/3070228206 >> v1:172.21.15.177:6789/0 conn(0x7f4bbc142710 legacy=0x7f4bbc14c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:42.646 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.658+0000 7f4bba7fc640 1 -- 172.21.15.7:0/3070228206 >> v1:172.21.15.7:6789/0 conn(0x7f4bbc0a8a60 legacy=0x7f4bbc13f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:42.646 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.658+0000 7f4bba7fc640 10 monclient: _finish_hunting 0 2024-01-21T03:19:42.646 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.658+0000 7f4bba7fc640 1 monclient: found mon.noname-a 2024-01-21T03:19:42.646 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.658+0000 7f4bba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:42.647 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.658+0000 7f4bba7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:19:42.647 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.658+0000 7f4bba7fc640 1 -- 172.21.15.7:0/3070228206 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc0aa6f0 con 0x7f4bbc13fa10 2024-01-21T03:19:42.647 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.661+0000 7f4bba7fc640 1 --2- 172.21.15.7:0/3070228206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bbc13fa10 0x7f4bbc1421d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4bb00097a0 tx=0x7f4bb002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88f08770ab021e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:42.656 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bb9ffb640 1 -- 172.21.15.7:0/3070228206 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0031830 con 0x7f4bbc13fa10 2024-01-21T03:19:42.656 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:42.656 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bb9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:19:42.656 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bb9ffb640 10 monclient: dump: 2024-01-21T03:19:42.657 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:42.657 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:42.657 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:42.657 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:42.657 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:42.657 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:42.657 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:42.657 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:42.657 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:42.658 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:42.658 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bb9ffb640 10 monclient: _finish_auth 0 2024-01-21T03:19:42.658 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bb9ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:19:42.658 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:42.658 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bb9ffb640 1 -- 172.21.15.7:0/3070228206 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb0031990 con 0x7f4bbc13fa10 2024-01-21T03:19:42.658 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bc1741640 5 monclient: authenticate success, global_id 4221 2024-01-21T03:19:42.658 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bc1741640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:19:42.658 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:42.659 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bc1741640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:42.659 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bb9ffb640 1 -- 172.21.15.7:0/3070228206 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0031d20 con 0x7f4bbc13fa10 2024-01-21T03:19:42.659 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bc1741640 4 set_mon_vals no callback set 2024-01-21T03:19:42.659 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:42.659 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bb9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:19:42.659 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.668+0000 7f4bb9ffb640 10 monclient: dump: 2024-01-21T03:19:42.659 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:42.660 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:42.660 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:42.660 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:42.660 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:42.660 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:42.660 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:42.660 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:42.661 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:42.661 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:42.661 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.669+0000 7f4bc1741640 10 monclient: shutdown 2024-01-21T03:19:42.661 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.669+0000 7f4bc1741640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:42.661 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.669+0000 7f4bc1741640 1 -- 172.21.15.7:0/3070228206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bbc13fa10 msgr2=0x7f4bbc1421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:42.661 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.669+0000 7f4bc1741640 1 --2- 172.21.15.7:0/3070228206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bbc13fa10 0x7f4bbc1421d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4bb00097a0 tx=0x7f4bb002b8d0 comp rx=0 tx=0).stop 2024-01-21T03:19:42.662 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.669+0000 7f4bc1741640 1 -- 172.21.15.7:0/3070228206 shutdown_connections 2024-01-21T03:19:42.662 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.669+0000 7f4bc1741640 1 --2- 172.21.15.7:0/3070228206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bbc13fa10 0x7f4bbc1421d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:42.662 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.669+0000 7f4bc1741640 1 -- 172.21.15.7:0/3070228206 >> 172.21.15.7:0/3070228206 conn(0x7f4bbc0670b0 msgr2=0x7f4bbc0b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:42.662 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.669+0000 7f4bc1741640 1 -- 172.21.15.7:0/3070228206 shutdown_connections 2024-01-21T03:19:42.662 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.669+0000 7f4bc1741640 1 -- 172.21.15.7:0/3070228206 wait complete. 2024-01-21T03:19:42.662 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.670+0000 7f4bc1741640 10 monclient: build_initial_monmap 2024-01-21T03:19:42.662 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.670+0000 7f4bc1741640 10 monclient: monmap: 2024-01-21T03:19:42.663 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:42.663 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:42.663 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-01-21T03:19:42.663 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-01-21T03:19:42.663 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:42.663 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:42.663 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:42.663 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:42.663 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:42.663 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:42.666 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.671+0000 7f4bc1741640 1 librados: starting msgr at 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.671+0000 7f4bc1741640 1 librados: starting objecter 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.671+0000 7f4bc1741640 1 Processor -- start 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.671+0000 7f4bc1741640 1 -- start start 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.671+0000 7f4bc1741640 1 librados: setting wanted keys 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.671+0000 7f4bc1741640 1 librados: calling monclient init 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.671+0000 7f4bc1741640 10 monclient: init 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bbc13fa10 0x7f4bbc135e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 10 monclient(hunting): picked mon.noname-a con 0x7f4bbc13fa10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4bbc142710 0x7f4bbc1daa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 10 monclient(hunting): picked mon.noname-c con 0x7f4bbc142710 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bbc136420 0x7f4bbc1dafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 10 monclient(hunting): picked mon.noname-b con 0x7f4bbc136420 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:42.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:42.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc1443b0 con 0x7f4bbc13fa10 2024-01-21T03:19:42.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:42.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc144600 con 0x7f4bbc136420 2024-01-21T03:19:42.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:42.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc1368b0 con 0x7f4bbc142710 2024-01-21T03:19:42.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:42.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.672+0000 7f4bc1741640 10 monclient(hunting): authenticate will time out at 1328.988257s 2024-01-21T03:19:42.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bbaffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bbc13fa10 0x7f4bbc135e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:42.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bbb7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bbc136420 0x7f4bbc1dafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:42.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bba7fc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4bbc142710 0x7f4bbc1daa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:42.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bbaffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bbc13fa10 0x7f4bbc135e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51484/0 (socket says 172.21.15.7:51484) 2024-01-21T03:19:42.668 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bbaffd640 1 -- 172.21.15.7:0/1063541941 learned_addr learned my addr 172.21.15.7:0/1063541941 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bbaffd640 10 monclient(hunting): get_auth_request con 0x7f4bbc13fa10 auth_method 0 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bbaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bbaffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bbb7fe640 10 monclient(hunting): get_auth_request con 0x7f4bbc136420 auth_method 0 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bbb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bba7fc640 10 monclient(hunting): get_auth_request con 0x7f4bbc142710 auth_method 0 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bba7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.673+0000 7f4bba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.674+0000 7f4bbaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.674+0000 7f4bbaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.674+0000 7f4bbaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.674+0000 7f4bbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.674+0000 7f4bbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.674+0000 7f4bbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.674+0000 7f4bba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.674+0000 7f4bba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.674+0000 7f4bba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.677+0000 7f4bbaffd640 10 monclient(hunting): handle_auth_done global_id 4227 payload 824 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.677+0000 7f4bbaffd640 1 -- 172.21.15.7:0/1063541941 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4bbc142710 msgr2=0x7f4bbc1daa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.677+0000 7f4bbaffd640 1 --2- 172.21.15.7:0/1063541941 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4bbc142710 0x7f4bbc1daa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.677+0000 7f4bbaffd640 1 -- 172.21.15.7:0/1063541941 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bbc136420 msgr2=0x7f4bbc1dafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.677+0000 7f4bbaffd640 1 --2- 172.21.15.7:0/1063541941 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bbc136420 0x7f4bbc1dafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.677+0000 7f4bbaffd640 10 monclient: _finish_hunting 0 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.677+0000 7f4bbaffd640 1 monclient: found mon.noname-a 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.677+0000 7f4bbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:42.670 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.677+0000 7f4bbaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:19:42.671 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.677+0000 7f4bbaffd640 1 -- 172.21.15.7:0/1063541941 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb0009400 con 0x7f4bbc13fa10 2024-01-21T03:19:42.671 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.679+0000 7f4bbaffd640 1 --2- 172.21.15.7:0/1063541941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bbc13fa10 0x7f4bbc135e90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00c400 tx=0x7f4bac00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:42.671 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.679+0000 7f4ba37fe640 10 client.?.objecter ms_handle_connect 0x7f4bbc13fa10 2024-01-21T03:19:42.671 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.679+0000 7f4ba37fe640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:42.672 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4ba37fe640 1 -- 172.21.15.7:0/1063541941 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0052d0 con 0x7f4bbc13fa10 2024-01-21T03:19:42.672 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4ba37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:42.672 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4ba37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:19:42.672 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4ba37fe640 10 monclient: dump: 2024-01-21T03:19:42.673 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:42.673 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:42.673 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:42.673 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:42.673 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:42.673 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:42.673 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:42.673 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:42.673 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:42.674 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:42.674 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4ba37fe640 10 monclient: _finish_auth 0 2024-01-21T03:19:42.674 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4ba37fe640 10 monclient: _check_auth_tickets 2024-01-21T03:19:42.674 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4ba37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:42.674 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4ba37fe640 1 -- 172.21.15.7:0/1063541941 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bac005450 con 0x7f4bbc13fa10 2024-01-21T03:19:42.674 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4bc1741640 5 monclient: authenticate success, global_id 4227 2024-01-21T03:19:42.674 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4ba37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:42.674 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4ba37fe640 1 -- 172.21.15.7:0/1063541941 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac007260 con 0x7f4bbc13fa10 2024-01-21T03:19:42.674 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4ba37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:42.674 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4ba37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:19:42.674 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4ba37fe640 10 monclient: dump: 2024-01-21T03:19:42.674 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:42.674 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.685+0000 7f4bb97fa640 4 set_mon_vals no callback set 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.686+0000 7f4bc1741640 10 monclient: _renew_subs 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.686+0000 7f4bc1741640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.686+0000 7f4bc1741640 1 -- 172.21.15.7:0/1063541941 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bbc136dd0 con 0x7f4bbc13fa10 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.686+0000 7f4bc1741640 10 client.4227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.686+0000 7f4bc1741640 10 monclient: _renew_subs 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.686+0000 7f4bc1741640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.686+0000 7f4bc1741640 1 -- 172.21.15.7:0/1063541941 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bbc143580 con 0x7f4bbc13fa10 2024-01-21T03:19:42.676 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.686+0000 7f4bc1741640 1 librados: init done 2024-01-21T03:19:42.676 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.686+0000 7f4bc1741640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:42.676 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.686+0000 7f4bc1741640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:42.676 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.686+0000 7f4bc1741640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:42.676 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.686+0000 7f4bc1741640 1 -- 172.21.15.7:0/1063541941 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bbc075c30 con 0x7f4bbc13fa10 2024-01-21T03:19:42.708 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.721+0000 7f4ba37fe640 1 -- 172.21.15.7:0/1063541941 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) v1 ==== 89142+0+0 (secure 0 0 0) 0x7f4bac0073c0 con 0x7f4bbc13fa10 2024-01-21T03:19:42.708 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.721+0000 7f4ba37fe640 1 --2- 172.21.15.7:0/1063541941 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4ba806c7d0 0x7f4ba806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:42.708 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.722+0000 7f4bba7fc640 1 --2- 172.21.15.7:0/1063541941 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4ba806c7d0 0x7f4ba806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:42.709 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.722+0000 7f4bba7fc640 10 monclient: get_auth_request con 0x7f4ba806c7d0 auth_method 0 2024-01-21T03:19:42.709 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.723+0000 7f4bba7fc640 1 --2- 172.21.15.7:0/1063541941 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4ba806c7d0 0x7f4ba806ec90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0036000 tx=0x7f4bb003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:42.709 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.723+0000 7f4ba37fe640 10 client.4227.objecter ms_handle_connect 0x7f4ba806c7d0 2024-01-21T03:19:42.970 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.983+0000 7f4ba37fe640 1 -- 172.21.15.7:0/1063541941 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2775+0+0 (secure 0 0 0) 0x7f4bac007670 con 0x7f4bbc13fa10 2024-01-21T03:19:42.970 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.983+0000 7f4ba37fe640 10 client.4227.objecter ms_dispatch 0x7f4bbc000b90 osd_map(14..14 src has 1..14) v4 2024-01-21T03:19:42.970 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.983+0000 7f4ba37fe640 3 client.4227.objecter handle_osd_map got epochs [14,14] > 0 2024-01-21T03:19:42.970 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.983+0000 7f4ba37fe640 3 client.4227.objecter handle_osd_map decoding full epoch 14 2024-01-21T03:19:42.970 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.984+0000 7f4ba37fe640 20 client.4227.objecter dump_active .. 0 homeless 2024-01-21T03:19:42.970 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.984+0000 7f4ba37fe640 1 -- 172.21.15.7:0/1063541941 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4bac0541d0 con 0x7f4bbc13fa10 2024-01-21T03:19:42.970 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.984+0000 7f4ba37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:42.970 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:42.984+0000 7f4ba37fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:43.213 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.227+0000 7f4bc1741640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:43.213 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.227+0000 7f4bc1741640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:43.213 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.227+0000 7f4bc1741640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:19:43.213 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.227+0000 7f4bc1741640 1 -- 172.21.15.7:0/1063541941 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bbc068300 con 0x7f4bbc13fa10 2024-01-21T03:19:43.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.319+0000 7f4ba37fe640 1 -- 172.21.15.7:0/1063541941 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+8053 (secure 0 0 0) 0x7f4bac020090 con 0x7f4bbc13fa10 2024-01-21T03:19:43.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.319+0000 7f4ba37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:43.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.319+0000 7f4ba37fe640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:43.306 INFO:teuthology.misc.health.smithi007.stdout: 2024-01-21T03:19:43.306 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":14,"fsid":"8de9e7dd-e982-4048-83dd-ee52350dd56d","created":"2024-01-21T03:18:57.607470+0000","modified":"2024-01-21T03:19:42.007762+0000","last_up_change":"2024-01-21T03:19:42.007762+0000","last_in_change":"2024-01-21T03:19:02.952334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f4767aa-192e-4ec9-addf-a51c0872da7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"066d7378-561f-4e74-b97b-2ead2fa2625e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f219649b-4d50-40a8-b74e-5a7f16b3c8c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6379a402-760a-4f07-9491-0918752a18fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dc3602e1-b59e-4489-9882-40c914e2a0d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6809","nonce":3975537014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6811","nonce":3975537014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6815","nonce":3975537014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6813","nonce":3975537014}]},"public_addr":"172.21.15.177:6809/3975537014","cluster_addr":"172.21.15.177:6811/3975537014","heartbeat_back_addr":"172.21.15.177:6815/3975537014","heartbeat_front_addr":"172.21.15.177:6813/3975537014","state":["exists","up"]},{"osd":5,"uuid":"d42afb34-e1ca-4b21-81e9-78d1c8ea454e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1aef2850-8b41-4d39-859e-ed9c4df434b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6823","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6825","nonce":4069998673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6827","nonce":4069998673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6831","nonce":4069998673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6829","nonce":4069998673}]},"public_addr":"172.21.15.177:6825/4069998673","cluster_addr":"172.21.15.177:6827/4069998673","heartbeat_back_addr":"172.21.15.177:6831/4069998673","heartbeat_front_addr":"172.21.15.177:6829/4069998673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T03:19:43.308 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.322+0000 7f4bc1741640 10 librados: watch_flush enter 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.322+0000 7f4bc1741640 10 librados: watch_flush exit 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.322+0000 7f4bc1741640 20 client.4227.objecter shutdown clearing up homeless session... 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.322+0000 7f4bc1741640 10 client.4227.objecter successfully canceled tick 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.322+0000 7f4bc1741640 1 -- 172.21.15.7:0/1063541941 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4ba806c7d0 msgr2=0x7f4ba806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.322+0000 7f4bc1741640 1 --2- 172.21.15.7:0/1063541941 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4ba806c7d0 0x7f4ba806ec90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0036000 tx=0x7f4bb003c040 comp rx=0 tx=0).stop 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.322+0000 7f4bc1741640 10 monclient: shutdown 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.322+0000 7f4bc1741640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.322+0000 7f4bc1741640 1 -- 172.21.15.7:0/1063541941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bbc13fa10 msgr2=0x7f4bbc135e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.322+0000 7f4bc1741640 1 --2- 172.21.15.7:0/1063541941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bbc13fa10 0x7f4bbc135e90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00c400 tx=0x7f4bac00c8c0 comp rx=0 tx=0).stop 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.323+0000 7f4bc1741640 1 -- 172.21.15.7:0/1063541941 shutdown_connections 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.323+0000 7f4bc1741640 1 --2- 172.21.15.7:0/1063541941 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4ba806c7d0 0x7f4ba806ec90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.323+0000 7f4bc1741640 1 --2- 172.21.15.7:0/1063541941 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bbc136420 0x7f4bbc1dafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.323+0000 7f4bc1741640 1 --2- 172.21.15.7:0/1063541941 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4bbc142710 0x7f4bbc1daa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:43.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.323+0000 7f4bc1741640 1 --2- 172.21.15.7:0/1063541941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bbc13fa10 0x7f4bbc135e90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:43.310 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.323+0000 7f4bc1741640 1 -- 172.21.15.7:0/1063541941 >> 172.21.15.7:0/1063541941 conn(0x7f4bbc0a8a60 msgr2=0x7f4bbc0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:43.310 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.323+0000 7f4bc1741640 1 -- 172.21.15.7:0/1063541941 shutdown_connections 2024-01-21T03:19:43.310 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.323+0000 7f4bc1741640 1 -- 172.21.15.7:0/1063541941 wait complete. 2024-01-21T03:19:43.310 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:43.323+0000 7f4bc1741640 1 librados: shutdown 2024-01-21T03:19:43.319 DEBUG:teuthology.misc:2 of 8 OSDs are up 2024-01-21T03:19:47.245 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:47.259+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 2 2024-01-21T03:19:47.379 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:47.320+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-01-21T03:19:47.389 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:47.393+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-01-21T03:19:47.393 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:47.247+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T03:19:47.497 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:47.408+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T03:19:47.577 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:47.581+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T03:19:47.631 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:47.592+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T03:19:47.631 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:47.603+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T03:19:47.641 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:47.645+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T03:19:47.668 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:47.656+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-01-21T03:19:47.728 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:47.406+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-01-21T03:19:48.084 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:47.684+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T03:19:48.172 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:47.667+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-01-21T03:19:48.172 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:47.745+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-01-21T03:19:48.173 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:47.939+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-01-21T03:19:48.173 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:48.035+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-01-21T03:19:48.173 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:48.101+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-01-21T03:19:48.179 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:48.188+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-01-21T03:19:48.207 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:47.750+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T03:19:48.212 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:47.956+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T03:19:49.121 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:49.129+0000 c493640 -1 osd.6 0 waiting for initial osdmap 2024-01-21T03:19:49.200 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:49.211+0000 c493640 -1 osd.5 0 waiting for initial osdmap 2024-01-21T03:19:49.606 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:19:49.606+0000 1425a640 -1 osd.6 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T03:19:49.619 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:19:49.620+0000 1425a640 -1 osd.5 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T03:19:50.126 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T03:19:50.279 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.292+0000 7f0c73861640 10 monclient: get_monmap_and_config 2024-01-21T03:19:50.279 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.292+0000 7f0c73861640 10 monclient: build_initial_monmap 2024-01-21T03:19:50.279 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.292+0000 7f0c73861640 10 monclient: monmap: 2024-01-21T03:19:50.279 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:50.279 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:50.279 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:19:50.293927+0000 2024-01-21T03:19:50.279 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:19:50.293927+0000 2024-01-21T03:19:50.279 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:50.279 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:50.279 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:50.280 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:50.280 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:50.280 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:50.280 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:50.280 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:50.283 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.293+0000 7f0c73861640 1 Processor -- start 2024-01-21T03:19:50.283 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.294+0000 7f0c73861640 1 -- start start 2024-01-21T03:19:50.283 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.294+0000 7f0c73861640 10 monclient: init 2024-01-21T03:19:50.283 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:50.284 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-01-21T03:19:50.284 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c6c0a8a60 0x7f0c6c13f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:50.284 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 10 monclient(hunting): picked mon.noname-b con 0x7f0c6c0a8a60 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:50.284 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0c6c13f840 0x7f0c6c1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:50.284 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 10 monclient(hunting): picked mon.noname-a con 0x7f0c6c13f840 addr v2:172.21.15.7:3300/0 2024-01-21T03:19:50.284 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0c6c142710 0x7f0c6c142b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:50.284 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 10 monclient(hunting): picked mon.noname-c con 0x7f0c6c142710 addr v2:172.21.15.177:3300/0 2024-01-21T03:19:50.284 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:50.284 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c6c0a8e40 con 0x7f0c6c13f840 2024-01-21T03:19:50.284 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c6c0b1de0 con 0x7f0c6c142710 2024-01-21T03:19:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c0aa480 con 0x7f0c6c0a8a60 2024-01-21T03:19:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.295+0000 7f0c73861640 10 monclient(hunting): authenticate will time out at 1336.610929s 2024-01-21T03:19:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.296+0000 7f0c71dfe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0c6c142710 0x7f0c6c142b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.296+0000 7f0c71dfe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0c6c142710 0x7f0c6c142b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:56230/0 (socket says 172.21.15.7:56230) 2024-01-21T03:19:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.296+0000 7f0c71dfe640 1 -- 172.21.15.7:0/2907225007 learned_addr learned my addr 172.21.15.7:0/2907225007 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.296+0000 7f0c71dfe640 10 monclient(hunting): get_auth_request con 0x7f0c6c142710 auth_method 0 2024-01-21T03:19:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.296+0000 7f0c71dfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.296+0000 7f0c71dfe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.296+0000 7f0c71dfe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:50.286 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.297+0000 7f0c71dfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:50.286 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.297+0000 7f0c71dfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:50.286 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.297+0000 7f0c71dfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:50.287 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.298+0000 7f0c71dfe640 10 monclient(hunting): handle_auth_done global_id 4240 payload 274 2024-01-21T03:19:50.287 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.298+0000 7f0c71dfe640 1 -- 172.21.15.7:0/2907225007 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c6c0a8a60 msgr2=0x7f0c6c13f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:50.287 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.298+0000 7f0c71dfe640 1 --2- 172.21.15.7:0/2907225007 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c6c0a8a60 0x7f0c6c13f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:50.288 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.298+0000 7f0c71dfe640 1 -- 172.21.15.7:0/2907225007 >> v2:172.21.15.7:3300/0 conn(0x7f0c6c13f840 msgr2=0x7f0c6c1421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:50.288 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.298+0000 7f0c71dfe640 1 --2- 172.21.15.7:0/2907225007 >> v2:172.21.15.7:3300/0 conn(0x7f0c6c13f840 0x7f0c6c1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:50.288 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.298+0000 7f0c71dfe640 10 monclient: _finish_hunting 0 2024-01-21T03:19:50.288 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.298+0000 7f0c71dfe640 1 monclient: found mon.noname-c 2024-01-21T03:19:50.288 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.298+0000 7f0c71dfe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:50.288 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.298+0000 7f0c71dfe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:19:50.288 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.298+0000 7f0c71dfe640 1 -- 172.21.15.7:0/2907225007 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c13fc40 con 0x7f0c6c142710 2024-01-21T03:19:50.288 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.300+0000 7f0c71dfe640 1 --2- 172.21.15.7:0/2907225007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c6c142710 0x7f0c6c142b40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c06bca0 tx=0x7f0c6402be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3052340c0865e455 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:50.296 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c63fff640 1 -- 172.21.15.7:0/2907225007 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6402c9f0 con 0x7f0c6c142710 2024-01-21T03:19:50.296 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:50.296 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c63fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:19:50.296 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c63fff640 10 monclient: dump: 2024-01-21T03:19:50.296 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:50.296 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:50.296 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:50.296 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:50.297 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:50.297 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:50.297 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:50.297 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:50.297 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:50.297 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:50.297 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c63fff640 10 monclient: _finish_auth 0 2024-01-21T03:19:50.297 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c63fff640 10 monclient: _check_auth_tickets 2024-01-21T03:19:50.297 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:50.297 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c63fff640 1 -- 172.21.15.7:0/2907225007 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6402cb50 con 0x7f0c6c142710 2024-01-21T03:19:50.297 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c73861640 5 monclient: authenticate success, global_id 4240 2024-01-21T03:19:50.297 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c73861640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:19:50.297 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c63fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c63fff640 1 -- 172.21.15.7:0/2907225007 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c640343f0 con 0x7f0c6c142710 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c63fff640 10 monclient: dump: 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:50.298 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c73861640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.304+0000 7f0c73861640 4 set_mon_vals no callback set 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.305+0000 7f0c73861640 10 monclient: shutdown 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.305+0000 7f0c73861640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.305+0000 7f0c73861640 1 -- 172.21.15.7:0/2907225007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c6c142710 msgr2=0x7f0c6c142b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.305+0000 7f0c73861640 1 --2- 172.21.15.7:0/2907225007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c6c142710 0x7f0c6c142b40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c06bca0 tx=0x7f0c6402be50 comp rx=0 tx=0).stop 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.305+0000 7f0c73861640 1 -- 172.21.15.7:0/2907225007 shutdown_connections 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.305+0000 7f0c73861640 1 --2- 172.21.15.7:0/2907225007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c6c142710 0x7f0c6c142b40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.305+0000 7f0c73861640 1 --2- 172.21.15.7:0/2907225007 >> v2:172.21.15.7:3300/0 conn(0x7f0c6c13f840 0x7f0c6c1421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.305+0000 7f0c73861640 1 --2- 172.21.15.7:0/2907225007 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c6c0a8a60 0x7f0c6c13f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.305+0000 7f0c73861640 1 -- 172.21.15.7:0/2907225007 >> 172.21.15.7:0/2907225007 conn(0x7f0c6c0670b0 msgr2=0x7f0c6c0b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.305+0000 7f0c73861640 1 -- 172.21.15.7:0/2907225007 shutdown_connections 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.305+0000 7f0c73861640 1 -- 172.21.15.7:0/2907225007 wait complete. 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.306+0000 7f0c73861640 10 monclient: build_initial_monmap 2024-01-21T03:19:50.299 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.306+0000 7f0c73861640 10 monclient: monmap: 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.307+0000 7f0c73861640 1 librados: starting msgr at 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.307+0000 7f0c73861640 1 librados: starting objecter 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.307+0000 7f0c73861640 1 Processor -- start 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.307+0000 7f0c73861640 1 -- start start 2024-01-21T03:19:50.300 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.307+0000 7f0c73861640 1 librados: setting wanted keys 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.307+0000 7f0c73861640 1 librados: calling monclient init 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.307+0000 7f0c73861640 10 monclient: init 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c6c13f840 0x7f0c6c1da980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 10 monclient(hunting): picked mon.noname-a con 0x7f0c6c13f840 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c6c142710 0x7f0c6c1daec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 10 monclient(hunting): picked mon.noname-b con 0x7f0c6c142710 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c6c0b6b90 0x7f0c6c0b7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 10 monclient(hunting): picked mon.noname-c con 0x7f0c6c0b6b90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c0b6250 con 0x7f0c6c13f840 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c1db400 con 0x7f0c6c142710 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:50.301 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c0b6700 con 0x7f0c6c0b6b90 2024-01-21T03:19:50.302 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:50.302 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c73861640 10 monclient(hunting): authenticate will time out at 1336.624166s 2024-01-21T03:19:50.302 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.308+0000 7f0c70dfc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c6c142710 0x7f0c6c1daec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:50.302 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.309+0000 7f0c70dfc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c6c142710 0x7f0c6c1daec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:56232/0 (socket says 172.21.15.7:56232) 2024-01-21T03:19:50.302 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.309+0000 7f0c70dfc640 1 -- 172.21.15.7:0/2282453327 learned_addr learned my addr 172.21.15.7:0/2282453327 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:50.302 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.309+0000 7f0c70dfc640 10 monclient(hunting): get_auth_request con 0x7f0c6c142710 auth_method 0 2024-01-21T03:19:50.302 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.309+0000 7f0c70dfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:50.302 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.309+0000 7f0c70dfc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:50.302 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.309+0000 7f0c70dfc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:50.302 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.309+0000 7f0c70dfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:50.302 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.309+0000 7f0c70dfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:50.302 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.309+0000 7f0c70dfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:50.305 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.310+0000 7f0c715fd640 1 --2- 172.21.15.7:0/2282453327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c6c13f840 0x7f0c6c1da980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:50.305 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.311+0000 7f0c715fd640 10 monclient(hunting): get_auth_request con 0x7f0c6c13f840 auth_method 0 2024-01-21T03:19:50.305 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.311+0000 7f0c715fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:50.305 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.311+0000 7f0c715fd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:50.305 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.311+0000 7f0c715fd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:50.305 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.311+0000 7f0c715fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:50.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.311+0000 7f0c715fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:50.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.311+0000 7f0c715fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:50.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.312+0000 7f0c70dfc640 10 monclient(hunting): handle_auth_done global_id 4243 payload 824 2024-01-21T03:19:50.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.312+0000 7f0c70dfc640 1 -- 172.21.15.7:0/2282453327 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c6c0b6b90 msgr2=0x7f0c6c0b7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:50.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.312+0000 7f0c70dfc640 1 --2- 172.21.15.7:0/2282453327 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c6c0b6b90 0x7f0c6c0b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:50.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.312+0000 7f0c70dfc640 1 -- 172.21.15.7:0/2282453327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c6c13f840 msgr2=0x7f0c6c1da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:50.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.312+0000 7f0c70dfc640 1 --2- 172.21.15.7:0/2282453327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c6c13f840 0x7f0c6c1da980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:50.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.312+0000 7f0c70dfc640 10 monclient: _finish_hunting 0 2024-01-21T03:19:50.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.312+0000 7f0c70dfc640 1 monclient: found mon.noname-b 2024-01-21T03:19:50.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.312+0000 7f0c70dfc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:50.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.312+0000 7f0c70dfc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:19:50.306 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.312+0000 7f0c70dfc640 1 -- 172.21.15.7:0/2282453327 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c64009b30 con 0x7f0c6c142710 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.314+0000 7f0c70dfc640 1 --2- 172.21.15.7:0/2282453327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c6c142710 0x7f0c6c1daec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800b4a0 tx=0x7f0c6800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.314+0000 7f0c61ffb640 10 client.?.objecter ms_handle_connect 0x7f0c6c142710 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.314+0000 7f0c61ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c61ffb640 1 -- 172.21.15.7:0/2282453327 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68011830 con 0x7f0c6c142710 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c61ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c61ffb640 10 monclient: dump: 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:50.307 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c61ffb640 10 monclient: _finish_auth 0 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c61ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c61ffb640 1 -- 172.21.15.7:0/2282453327 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c68011990 con 0x7f0c6c142710 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c61ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c61ffb640 1 -- 172.21.15.7:0/2282453327 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68011d40 con 0x7f0c6c142710 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c61ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c61ffb640 10 monclient: dump: 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:50.308 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c73861640 5 monclient: authenticate success, global_id 4243 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c73861640 10 monclient: _renew_subs 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c73861640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.317+0000 7f0c73861640 1 -- 172.21.15.7:0/2282453327 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6c144100 con 0x7f0c6c142710 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.318+0000 7f0c62ffd640 4 set_mon_vals no callback set 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.318+0000 7f0c73861640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.318+0000 7f0c73861640 10 monclient: _renew_subs 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.318+0000 7f0c73861640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.318+0000 7f0c73861640 1 -- 172.21.15.7:0/2282453327 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6c0b7580 con 0x7f0c6c142710 2024-01-21T03:19:50.309 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.318+0000 7f0c73861640 1 librados: init done 2024-01-21T03:19:50.310 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.320+0000 7f0c73861640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:50.310 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.320+0000 7f0c73861640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:50.310 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.320+0000 7f0c73861640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:50.310 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.320+0000 7f0c73861640 1 -- 172.21.15.7:0/2282453327 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6c06b500 con 0x7f0c6c142710 2024-01-21T03:19:50.323 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.337+0000 7f0c61ffb640 1 -- 172.21.15.7:0/2282453327 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) v1 ==== 89142+0+0 (secure 0 0 0) 0x7f0c6801a400 con 0x7f0c6c142710 2024-01-21T03:19:50.323 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.337+0000 7f0c61ffb640 1 --2- 172.21.15.7:0/2282453327 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f0c5006c7d0 0x7f0c5006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:50.324 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.337+0000 7f0c715fd640 1 --2- 172.21.15.7:0/2282453327 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f0c5006c7d0 0x7f0c5006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:50.324 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.337+0000 7f0c715fd640 10 monclient: get_auth_request con 0x7f0c5006c7d0 auth_method 0 2024-01-21T03:19:50.324 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.338+0000 7f0c715fd640 1 --2- 172.21.15.7:0/2282453327 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f0c5006c7d0 0x7f0c5006ec90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009990 tx=0x7f0c5c009110 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:50.324 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.338+0000 7f0c61ffb640 10 client.4243.objecter ms_handle_connect 0x7f0c5006c7d0 2024-01-21T03:19:50.548 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.561+0000 7f0c61ffb640 1 -- 172.21.15.7:0/2282453327 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3177+0+0 (secure 0 0 0) 0x7f0c68058fe0 con 0x7f0c6c142710 2024-01-21T03:19:50.548 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.561+0000 7f0c61ffb640 10 client.4243.objecter ms_dispatch 0x7f0c6c000b90 osd_map(17..17 src has 1..17) v4 2024-01-21T03:19:50.548 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.561+0000 7f0c61ffb640 3 client.4243.objecter handle_osd_map got epochs [17,17] > 0 2024-01-21T03:19:50.548 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.561+0000 7f0c61ffb640 3 client.4243.objecter handle_osd_map decoding full epoch 17 2024-01-21T03:19:50.548 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.561+0000 7f0c61ffb640 20 client.4243.objecter dump_active .. 0 homeless 2024-01-21T03:19:50.560 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.573+0000 7f0c61ffb640 1 -- 172.21.15.7:0/2282453327 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f0c68054c20 con 0x7f0c6c142710 2024-01-21T03:19:50.560 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.573+0000 7f0c61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:50.560 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.573+0000 7f0c61ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:50.938 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.951+0000 7f0c73861640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:50.938 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.951+0000 7f0c73861640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:50.939 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.951+0000 7f0c73861640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:50.939 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:50.951+0000 7f0c73861640 1 -- 172.21.15.7:0/2282453327 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c6c075c30 con 0x7f0c6c142710 2024-01-21T03:19:51.032 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.045+0000 7f0c61ffb640 1 -- 172.21.15.7:0/2282453327 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+10888 (secure 0 0 0) 0x7f0c68015020 con 0x7f0c6c142710 2024-01-21T03:19:51.032 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.045+0000 7f0c61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:51.033 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.045+0000 7f0c61ffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:51.033 INFO:teuthology.misc.health.smithi007.stdout: 2024-01-21T03:19:51.033 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":18,"fsid":"8de9e7dd-e982-4048-83dd-ee52350dd56d","created":"2024-01-21T03:18:57.607470+0000","modified":"2024-01-21T03:19:50.556903+0000","last_up_change":"2024-01-21T03:19:50.556903+0000","last_in_change":"2024-01-21T03:19:02.952334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-21T03:19:44.021407+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f4767aa-192e-4ec9-addf-a51c0872da7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"066d7378-561f-4e74-b97b-2ead2fa2625e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f219649b-4d50-40a8-b74e-5a7f16b3c8c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6379a402-760a-4f07-9491-0918752a18fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dc3602e1-b59e-4489-9882-40c914e2a0d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6809","nonce":3975537014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6811","nonce":3975537014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6815","nonce":3975537014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6813","nonce":3975537014}]},"public_addr":"172.21.15.177:6809/3975537014","cluster_addr":"172.21.15.177:6811/3975537014","heartbeat_back_addr":"172.21.15.177:6815/3975537014","heartbeat_front_addr":"172.21.15.177:6813/3975537014","state":["exists","up"]},{"osd":5,"uuid":"d42afb34-e1ca-4b21-81e9-78d1c8ea454e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6817","nonce":2318383718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6819","nonce":2318383718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6824","nonce":2318383718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6821","nonce":2318383718}]},"public_addr":"172.21.15.177:6817/2318383718","cluster_addr":"172.21.15.177:6819/2318383718","heartbeat_back_addr":"172.21.15.177:6824/2318383718","heartbeat_front_addr":"172.21.15.177:6821/2318383718","state":["exists","up"]},{"osd":6,"uuid":"1aef2850-8b41-4d39-859e-ed9c4df434b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6801","nonce":1092304531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6803","nonce":1092304531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6807","nonce":1092304531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6805","nonce":1092304531}]},"public_addr":"172.21.15.177:6801/1092304531","cluster_addr":"172.21.15.177:6803/1092304531","heartbeat_back_addr":"172.21.15.177:6807/1092304531","heartbeat_front_addr":"172.21.15.177:6805/1092304531","state":["exists","up"]},{"osd":7,"uuid":"8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6823","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6825","nonce":4069998673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6827","nonce":4069998673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6831","nonce":4069998673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6829","nonce":4069998673}]},"public_addr":"172.21.15.177:6825/4069998673","cluster_addr":"172.21.15.177:6827/4069998673","heartbeat_back_addr":"172.21.15.177:6831/4069998673","heartbeat_front_addr":"172.21.15.177:6829/4069998673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:38.955717+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:39.042445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T03:19:51.035 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.048+0000 7f0c73861640 10 librados: watch_flush enter 2024-01-21T03:19:51.037 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.048+0000 7f0c73861640 10 librados: watch_flush exit 2024-01-21T03:19:51.037 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.048+0000 7f0c73861640 20 client.4243.objecter shutdown clearing up homeless session... 2024-01-21T03:19:51.037 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.048+0000 7f0c73861640 10 client.4243.objecter successfully canceled tick 2024-01-21T03:19:51.037 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.048+0000 7f0c73861640 1 -- 172.21.15.7:0/2282453327 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f0c5006c7d0 msgr2=0x7f0c5006ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:51.037 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.048+0000 7f0c73861640 1 --2- 172.21.15.7:0/2282453327 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f0c5006c7d0 0x7f0c5006ec90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009990 tx=0x7f0c5c009110 comp rx=0 tx=0).stop 2024-01-21T03:19:51.037 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.048+0000 7f0c73861640 10 monclient: shutdown 2024-01-21T03:19:51.037 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.048+0000 7f0c73861640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:51.037 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.048+0000 7f0c73861640 1 -- 172.21.15.7:0/2282453327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c6c142710 msgr2=0x7f0c6c1daec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:51.037 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.048+0000 7f0c73861640 1 --2- 172.21.15.7:0/2282453327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c6c142710 0x7f0c6c1daec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800b4a0 tx=0x7f0c6800b8a0 comp rx=0 tx=0).stop 2024-01-21T03:19:51.038 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.049+0000 7f0c73861640 1 -- 172.21.15.7:0/2282453327 shutdown_connections 2024-01-21T03:19:51.038 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.049+0000 7f0c73861640 1 --2- 172.21.15.7:0/2282453327 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f0c5006c7d0 0x7f0c5006ec90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:51.038 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.049+0000 7f0c73861640 1 --2- 172.21.15.7:0/2282453327 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c6c0b6b90 0x7f0c6c0b7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:51.038 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.049+0000 7f0c73861640 1 --2- 172.21.15.7:0/2282453327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c6c142710 0x7f0c6c1daec0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800b4a0 tx=0x7f0c6800b8a0 comp rx=0 tx=0).stop 2024-01-21T03:19:51.038 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.049+0000 7f0c73861640 1 --2- 172.21.15.7:0/2282453327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c6c13f840 0x7f0c6c1da980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:51.038 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.049+0000 7f0c73861640 1 -- 172.21.15.7:0/2282453327 >> 172.21.15.7:0/2282453327 conn(0x7f0c6c0a8a60 msgr2=0x7f0c6c073df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:51.038 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.049+0000 7f0c73861640 1 -- 172.21.15.7:0/2282453327 shutdown_connections 2024-01-21T03:19:51.038 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.049+0000 7f0c73861640 1 -- 172.21.15.7:0/2282453327 wait complete. 2024-01-21T03:19:51.038 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:51.049+0000 7f0c73861640 1 librados: shutdown 2024-01-21T03:19:51.053 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-01-21T03:19:51.543 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:51.544+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-21T03:19:51.637 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:51.557+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-21T03:19:51.726 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:51.651+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-21T03:19:51.739 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:19:51.387+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-01-21T03:19:51.791 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:19:51.752+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-01-21T03:19:51.797 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:51.741+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-21T03:19:51.809 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:51.811+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-21T03:19:51.845 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:51.823+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-21T03:19:51.846 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:51.847+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-21T03:19:51.983 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:19:51.463+0000 5dc2300 -1 bdev(0x18e78000 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-01-21T03:19:52.013 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:19:51.810+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-01-21T03:19:52.020 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:51.861+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-21T03:19:52.021 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:51.891+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-21T03:19:52.238 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:52.160+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-21T03:19:52.303 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:52.263+0000 5dc2300 -1 bdev(0x18e701e0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-21T03:19:52.983 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:19:52.950+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-21T03:19:53.041 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:19:52.996+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-21T03:19:53.300 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:19:53.018+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-21T03:19:53.327 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:19:53.336+0000 ac7d640 -1 osd.0 0 waiting for initial osdmap 2024-01-21T03:19:53.387 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:19:53.395+0000 ac7d640 -1 osd.2 0 waiting for initial osdmap 2024-01-21T03:19:53.397 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:19:53.359+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-21T03:19:53.614 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:53.621+0000 c493640 -1 osd.1 0 waiting for initial osdmap 2024-01-21T03:19:53.870 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:19:53.867+0000 1425a640 -1 osd.0 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T03:19:53.924 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:19:53.921+0000 1425a640 -1 osd.2 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T03:19:54.136 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:19:54.134+0000 1425a640 -1 osd.1 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T03:19:54.768 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:19:54.778+0000 ac7d640 -1 osd.3 0 waiting for initial osdmap 2024-01-21T03:19:55.248 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:19:55.248+0000 1425a640 -1 osd.3 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T03:19:55.746 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:10.075 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:19:55.813 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:10.142 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:19:55.827 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:10.101 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:19:55.829 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:10.158 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:19:55.885 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:10.160 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:19:55.906 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:10.235 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:19:56.082 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:10.411 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:19:56.144 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:10.418 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:19:56.235 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:10.564 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:19:57.860 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T03:19:57.980 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.994+0000 7f622b0ec640 10 monclient: get_monmap_and_config 2024-01-21T03:19:57.980 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.994+0000 7f622b0ec640 10 monclient: build_initial_monmap 2024-01-21T03:19:57.980 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.994+0000 7f622b0ec640 10 monclient: monmap: 2024-01-21T03:19:57.980 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:57.980 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:57.980 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:19:57.995318+0000 2024-01-21T03:19:57.980 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:19:57.995318+0000 2024-01-21T03:19:57.980 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:57.981 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:57.981 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:57.981 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:57.981 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:57.981 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:57.981 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:57.981 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:57.983 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 1 Processor -- start 2024-01-21T03:19:57.983 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 1 -- start start 2024-01-21T03:19:57.983 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 10 monclient: init 2024-01-21T03:19:57.983 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:57.983 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-01-21T03:19:57.983 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62240a8a60 addr v1:172.21.15.177:6789/0 2024-01-21T03:19:57.983 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f622413fa10 0x7f62241421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:57.983 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 10 monclient(hunting): picked mon.noname-c con 0x7f622413fa10 addr v2:172.21.15.177:3300/0 2024-01-21T03:19:57.984 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6224142710 addr v1:172.21.15.7:6789/0 2024-01-21T03:19:57.984 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62240b1de0 con 0x7f6224142710 2024-01-21T03:19:57.984 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62240aa480 con 0x7f62240a8a60 2024-01-21T03:19:57.984 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:57.984 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62240a8e40 con 0x7f622413fa10 2024-01-21T03:19:57.984 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:57.984 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.995+0000 7f622b0ec640 10 monclient(hunting): authenticate will time out at 1344.311568s 2024-01-21T03:19:57.985 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.996+0000 7f6229689640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6224142710 0x7f622414c930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59790/0 (socket says 172.21.15.7:59790) 2024-01-21T03:19:57.985 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.996+0000 7f6229689640 1 -- 172.21.15.7:0/3848097004 learned_addr learned my addr 172.21.15.7:0/3848097004 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:19:57.985 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.996+0000 7f6223fff640 1 --2- 172.21.15.7:0/3848097004 >> v2:172.21.15.177:3300/0 conn(0x7f622413fa10 0x7f62241421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:57.985 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.997+0000 7f6223fff640 10 monclient(hunting): get_auth_request con 0x7f622413fa10 auth_method 0 2024-01-21T03:19:57.985 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.997+0000 7f6223fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:57.985 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.997+0000 7f6223fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:57.986 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.997+0000 7f6223fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:57.987 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.997+0000 7f6223fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:57.987 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.997+0000 7f6223fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:57.988 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.997+0000 7f6223fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:57.988 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f62237fe640 1 -- 172.21.15.7:0/3848097004 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785387753 0 0) 0x7f62240b1de0 con 0x7f6224142710 2024-01-21T03:19:57.988 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f62237fe640 10 monclient(hunting): _negotiate 2024-01-21T03:19:57.988 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f62237fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:57.988 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f62237fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:57.988 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f62237fe640 10 monclient(hunting): my global_id is 4242 2024-01-21T03:19:57.988 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f62237fe640 1 -- 172.21.15.7:0/3848097004 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6214003520 con 0x7f6224142710 2024-01-21T03:19:57.988 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f6223fff640 10 monclient(hunting): handle_auth_done global_id 4249 payload 274 2024-01-21T03:19:57.989 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f6223fff640 1 -- 172.21.15.7:0/3848097004 >> v1:172.21.15.177:6789/0 conn(0x7f62240a8a60 legacy=0x7f622413f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:57.989 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f6223fff640 1 -- 172.21.15.7:0/3848097004 >> v1:172.21.15.7:6789/0 conn(0x7f6224142710 legacy=0x7f622414c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:57.989 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f6223fff640 10 monclient: _finish_hunting 0 2024-01-21T03:19:57.989 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f6223fff640 1 monclient: found mon.noname-c 2024-01-21T03:19:57.989 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f6223fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:57.989 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f6223fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:19:57.989 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:57.999+0000 7f6223fff640 1 -- 172.21.15.7:0/3848097004 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62240aa6f0 con 0x7f622413fa10 2024-01-21T03:19:57.990 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.003+0000 7f6223fff640 1 --2- 172.21.15.7:0/3848097004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f622413fa10 0x7f62241421d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f62100097a0 tx=0x7f621002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10e473a949397e7a server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:57.993 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f62237fe640 1 -- 172.21.15.7:0/3848097004 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210031830 con 0x7f622413fa10 2024-01-21T03:19:57.993 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f62237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:57.993 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f62237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:19:57.993 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f62237fe640 10 monclient: dump: 2024-01-21T03:19:57.993 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:57.993 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:57.994 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:57.994 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:57.994 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:57.994 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:57.994 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:57.994 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:57.994 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:57.994 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:57.994 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f62237fe640 10 monclient: _finish_auth 0 2024-01-21T03:19:57.994 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f62237fe640 10 monclient: _check_auth_tickets 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f62237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f62237fe640 1 -- 172.21.15.7:0/3848097004 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6210031990 con 0x7f622413fa10 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f62237fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f62237fe640 1 -- 172.21.15.7:0/3848097004 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210031d20 con 0x7f622413fa10 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f62237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f62237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f62237fe640 10 monclient: dump: 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:57.995 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:57.996 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:57.996 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:57.996 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:57.996 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f622b0ec640 5 monclient: authenticate success, global_id 4249 2024-01-21T03:19:57.996 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f622b0ec640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:57.996 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.007+0000 7f622b0ec640 4 set_mon_vals no callback set 2024-01-21T03:19:58.000 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.008+0000 7f622b0ec640 10 monclient: shutdown 2024-01-21T03:19:58.000 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.008+0000 7f622b0ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:58.000 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.008+0000 7f622b0ec640 1 -- 172.21.15.7:0/3848097004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f622413fa10 msgr2=0x7f62241421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:58.000 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.008+0000 7f622b0ec640 1 --2- 172.21.15.7:0/3848097004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f622413fa10 0x7f62241421d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f62100097a0 tx=0x7f621002b8d0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.000 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.008+0000 7f622b0ec640 1 -- 172.21.15.7:0/3848097004 shutdown_connections 2024-01-21T03:19:58.000 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.008+0000 7f622b0ec640 1 --2- 172.21.15.7:0/3848097004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f622413fa10 0x7f62241421d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.001 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.008+0000 7f622b0ec640 1 -- 172.21.15.7:0/3848097004 >> 172.21.15.7:0/3848097004 conn(0x7f62240670b0 msgr2=0x7f62240b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:58.001 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.008+0000 7f622b0ec640 1 -- 172.21.15.7:0/3848097004 shutdown_connections 2024-01-21T03:19:58.001 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.008+0000 7f622b0ec640 1 -- 172.21.15.7:0/3848097004 wait complete. 2024-01-21T03:19:58.001 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.009+0000 7f622b0ec640 10 monclient: build_initial_monmap 2024-01-21T03:19:58.001 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.009+0000 7f622b0ec640 10 monclient: monmap: 2024-01-21T03:19:58.001 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-01-21T03:19:58.001 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:58.001 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-01-21T03:19:58.001 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-01-21T03:19:58.001 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:58.001 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:58.001 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:58.001 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.009+0000 7f622b0ec640 1 librados: starting msgr at 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.009+0000 7f622b0ec640 1 librados: starting objecter 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.009+0000 7f622b0ec640 1 Processor -- start 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.009+0000 7f622b0ec640 1 -- start start 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.009+0000 7f622b0ec640 1 librados: setting wanted keys 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.009+0000 7f622b0ec640 1 librados: calling monclient init 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.009+0000 7f622b0ec640 10 monclient: init 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f622413fa10 0x7f62241448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 10 monclient(hunting): picked mon.noname-c con 0x7f622413fa10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6224142710 0x7f622413b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 10 monclient(hunting): picked mon.noname-b con 0x7f6224142710 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:58.002 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6224145900 0x7f6224136a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 10 monclient(hunting): picked mon.noname-a con 0x7f6224145900 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6224144f50 con 0x7f6224145900 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62240abaa0 con 0x7f6224142710 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6224136f60 con 0x7f622413fa10 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f622b0ec640 10 monclient(hunting): authenticate will time out at 1344.326173s 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f6223fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6224142710 0x7f622413b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f6229689640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6224145900 0x7f6224136a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f6223fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6224142710 0x7f622413b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36440/0 (socket says 172.21.15.7:36440) 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f6223fff640 1 -- 172.21.15.7:0/2745227345 learned_addr learned my addr 172.21.15.7:0/2745227345 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f6223fff640 10 monclient(hunting): get_auth_request con 0x7f6224142710 auth_method 0 2024-01-21T03:19:58.003 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f6223fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:58.004 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f6223fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:58.004 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.010+0000 7f6223fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:58.004 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.011+0000 7f6229689640 10 monclient(hunting): get_auth_request con 0x7f6224145900 auth_method 0 2024-01-21T03:19:58.004 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.011+0000 7f6229689640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:58.004 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.011+0000 7f6229689640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:58.004 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.011+0000 7f6229689640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:58.004 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.011+0000 7f6228e88640 1 --2- 172.21.15.7:0/2745227345 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f622413fa10 0x7f62241448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:58.004 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.011+0000 7f6223fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:58.004 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.011+0000 7f6223fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:58.004 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.011+0000 7f6223fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:58.004 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.011+0000 7f6228e88640 10 monclient(hunting): get_auth_request con 0x7f622413fa10 auth_method 0 2024-01-21T03:19:58.004 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.011+0000 7f6228e88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:58.004 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.011+0000 7f6228e88640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.011+0000 7f6228e88640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.012+0000 7f6229689640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.012+0000 7f6229689640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.012+0000 7f6229689640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.012+0000 7f6228e88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.012+0000 7f6228e88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.012+0000 7f6228e88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.014+0000 7f6223fff640 10 monclient(hunting): handle_auth_done global_id 4252 payload 824 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.014+0000 7f6223fff640 1 -- 172.21.15.7:0/2745227345 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f622413fa10 msgr2=0x7f62241448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.014+0000 7f6223fff640 1 --2- 172.21.15.7:0/2745227345 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f622413fa10 0x7f62241448e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.014+0000 7f6223fff640 1 -- 172.21.15.7:0/2745227345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6224145900 msgr2=0x7f6224136a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.014+0000 7f6223fff640 1 --2- 172.21.15.7:0/2745227345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6224145900 0x7f6224136a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.014+0000 7f6223fff640 10 monclient: _finish_hunting 0 2024-01-21T03:19:58.005 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.014+0000 7f6223fff640 1 monclient: found mon.noname-b 2024-01-21T03:19:58.006 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.014+0000 7f6223fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:58.006 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.014+0000 7f6223fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:19:58.006 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.014+0000 7f6223fff640 1 -- 172.21.15.7:0/2745227345 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6210009400 con 0x7f6224142710 2024-01-21T03:19:58.007 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.016+0000 7f6223fff640 1 --2- 172.21.15.7:0/2745227345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6224142710 0x7f622413b800 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f621002bee0 tx=0x7f62100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:58.007 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.016+0000 7f62217fa640 10 client.?.objecter ms_handle_connect 0x7f6224142710 2024-01-21T03:19:58.007 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.016+0000 7f62217fa640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:58.007 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f62217fa640 1 -- 172.21.15.7:0/2745227345 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621003b9f0 con 0x7f6224142710 2024-01-21T03:19:58.007 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f62217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:58.007 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f62217fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:58.007 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f62217fa640 10 monclient: dump: 2024-01-21T03:19:58.007 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:58.007 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:58.007 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:58.008 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:58.008 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:58.008 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:58.008 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:58.008 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:58.008 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:58.008 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:58.008 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f62217fa640 10 monclient: _finish_auth 0 2024-01-21T03:19:58.008 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f62217fa640 10 monclient: _check_auth_tickets 2024-01-21T03:19:58.008 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f62217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:58.008 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f62217fa640 1 -- 172.21.15.7:0/2745227345 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621003bb70 con 0x7f6224142710 2024-01-21T03:19:58.008 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f62217fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:58.008 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f62217fa640 1 -- 172.21.15.7:0/2745227345 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62100453f0 con 0x7f6224142710 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f62217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f62217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f62217fa640 10 monclient: dump: 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr: 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f622b0ec640 5 monclient: authenticate success, global_id 4252 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f622b0ec640 10 monclient: _renew_subs 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f6222ffd640 4 set_mon_vals no callback set 2024-01-21T03:19:58.009 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f622b0ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:58.010 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f622b0ec640 1 -- 172.21.15.7:0/2745227345 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6224137180 con 0x7f6224142710 2024-01-21T03:19:58.010 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f622b0ec640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:58.010 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f622b0ec640 10 monclient: _renew_subs 2024-01-21T03:19:58.010 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f622b0ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:58.010 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f622b0ec640 1 -- 172.21.15.7:0/2745227345 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62241da7e0 con 0x7f6224142710 2024-01-21T03:19:58.010 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.019+0000 7f622b0ec640 1 librados: init done 2024-01-21T03:19:58.010 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.020+0000 7f622b0ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:58.010 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.020+0000 7f622b0ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:58.010 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.020+0000 7f622b0ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:58.010 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.020+0000 7f622b0ec640 1 -- 172.21.15.7:0/2745227345 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62241450c0 con 0x7f6224142710 2024-01-21T03:19:58.258 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.272+0000 7f62217fa640 1 -- 172.21.15.7:0/2745227345 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6210045550 con 0x7f6224142710 2024-01-21T03:19:58.258 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.272+0000 7f62217fa640 1 --2- 172.21.15.7:0/2745227345 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f620806c8e0 0x7f620806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:58.258 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.272+0000 7f6228e88640 1 --2- 172.21.15.7:0/2745227345 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f620806c8e0 0x7f620806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:58.259 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.272+0000 7f6228e88640 10 monclient: get_auth_request con 0x7f620806c8e0 auth_method 0 2024-01-21T03:19:58.259 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.273+0000 7f6228e88640 1 --2- 172.21.15.7:0/2745227345 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f620806c8e0 0x7f620806eda0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f62240b3570 tx=0x7f6218009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:58.259 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.273+0000 7f62217fa640 10 client.4252.objecter ms_handle_connect 0x7f620806c8e0 2024-01-21T03:19:58.262 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.276+0000 7f62217fa640 1 -- 172.21.15.7:0/2745227345 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f621007acf0 con 0x7f6224142710 2024-01-21T03:19:58.262 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.276+0000 7f62217fa640 10 client.4252.objecter ms_dispatch 0x7f6224000b90 osd_map(23..23 src has 1..23) v4 2024-01-21T03:19:58.262 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.276+0000 7f62217fa640 3 client.4252.objecter handle_osd_map got epochs [23,23] > 0 2024-01-21T03:19:58.262 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.276+0000 7f62217fa640 3 client.4252.objecter handle_osd_map decoding full epoch 23 2024-01-21T03:19:58.262 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.277+0000 7f62217fa640 20 client.4252.objecter dump_active .. 0 homeless 2024-01-21T03:19:58.277 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.291+0000 7f62217fa640 1 -- 172.21.15.7:0/2745227345 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6210077030 con 0x7f6224142710 2024-01-21T03:19:58.277 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.291+0000 7f62217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:58.277 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.291+0000 7f62217fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:58.461 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.476+0000 7f622b0ec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:58.462 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.476+0000 7f622b0ec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:58.462 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.476+0000 7f622b0ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:58.462 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.476+0000 7f622b0ec640 1 -- 172.21.15.7:0/2745227345 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6224075c30 con 0x7f6224142710 2024-01-21T03:19:58.488 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.502+0000 7f62217fa640 1 -- 172.21.15.7:0/2745227345 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+12804 (secure 0 0 0) 0x7f621007a650 con 0x7f6224142710 2024-01-21T03:19:58.488 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.502+0000 7f62217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:19:58.488 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.502+0000 7f62217fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:19:58.488 INFO:teuthology.misc.health.smithi007.stdout: 2024-01-21T03:19:58.489 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":23,"fsid":"8de9e7dd-e982-4048-83dd-ee52350dd56d","created":"2024-01-21T03:18:57.607470+0000","modified":"2024-01-21T03:19:57.625965+0000","last_up_change":"2024-01-21T03:19:56.402389+0000","last_in_change":"2024-01-21T03:19:02.952334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-21T03:19:44.021407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f4767aa-192e-4ec9-addf-a51c0872da7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6819","nonce":2019292672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6821","nonce":2019292672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6825","nonce":2019292672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6823","nonce":2019292672}]},"public_addr":"172.21.15.7:6819/2019292672","cluster_addr":"172.21.15.7:6821/2019292672","heartbeat_back_addr":"172.21.15.7:6825/2019292672","heartbeat_front_addr":"172.21.15.7:6823/2019292672","state":["exists","up"]},{"osd":1,"uuid":"066d7378-561f-4e74-b97b-2ead2fa2625e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6811","nonce":2176696455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6813","nonce":2176696455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6817","nonce":2176696455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6815","nonce":2176696455}]},"public_addr":"172.21.15.7:6811/2176696455","cluster_addr":"172.21.15.7:6813/2176696455","heartbeat_back_addr":"172.21.15.7:6817/2176696455","heartbeat_front_addr":"172.21.15.7:6815/2176696455","state":["exists","up"]},{"osd":2,"uuid":"f219649b-4d50-40a8-b74e-5a7f16b3c8c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6827","nonce":888884750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6829","nonce":888884750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6833","nonce":888884750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6831","nonce":888884750}]},"public_addr":"172.21.15.7:6827/888884750","cluster_addr":"172.21.15.7:6829/888884750","heartbeat_back_addr":"172.21.15.7:6833/888884750","heartbeat_front_addr":"172.21.15.7:6831/888884750","state":["exists","up"]},{"osd":3,"uuid":"6379a402-760a-4f07-9491-0918752a18fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6803","nonce":1182044719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6805","nonce":1182044719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6809","nonce":1182044719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6807","nonce":1182044719}]},"public_addr":"172.21.15.7:6803/1182044719","cluster_addr":"172.21.15.7:6805/1182044719","heartbeat_back_addr":"172.21.15.7:6809/1182044719","heartbeat_front_addr":"172.21.15.7:6807/1182044719","state":["exists","up"]},{"osd":4,"uuid":"dc3602e1-b59e-4489-9882-40c914e2a0d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6809","nonce":3975537014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6811","nonce":3975537014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6815","nonce":3975537014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6813","nonce":3975537014}]},"public_addr":"172.21.15.177:6809/3975537014","cluster_addr":"172.21.15.177:6811/3975537014","heartbeat_back_addr":"172.21.15.177:6815/3975537014","heartbeat_front_addr":"172.21.15.177:6813/3975537014","state":["exists","up"]},{"osd":5,"uuid":"d42afb34-e1ca-4b21-81e9-78d1c8ea454e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6817","nonce":2318383718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6819","nonce":2318383718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6824","nonce":2318383718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6821","nonce":2318383718}]},"public_addr":"172.21.15.177:6817/2318383718","cluster_addr":"172.21.15.177:6819/2318383718","heartbeat_back_addr":"172.21.15.177:6824/2318383718","heartbeat_front_addr":"172.21.15.177:6821/2318383718","state":["exists","up"]},{"osd":6,"uuid":"1aef2850-8b41-4d39-859e-ed9c4df434b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6801","nonce":1092304531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6803","nonce":1092304531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6807","nonce":1092304531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6805","nonce":1092304531}]},"public_addr":"172.21.15.177:6801/1092304531","cluster_addr":"172.21.15.177:6803/1092304531","heartbeat_back_addr":"172.21.15.177:6807/1092304531","heartbeat_front_addr":"172.21.15.177:6805/1092304531","state":["exists","up"]},{"osd":7,"uuid":"8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6823","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6825","nonce":4069998673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6827","nonce":4069998673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6831","nonce":4069998673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6829","nonce":4069998673}]},"public_addr":"172.21.15.177:6825/4069998673","cluster_addr":"172.21.15.177:6827/4069998673","heartbeat_back_addr":"172.21.15.177:6831/4069998673","heartbeat_front_addr":"172.21.15.177:6829/4069998673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:40.924122+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:41.225506+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:41.727729+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:41.835880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:38.955717+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:38.638101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:39.011972+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:39.042445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 10 librados: watch_flush enter 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 10 librados: watch_flush exit 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 20 client.4252.objecter shutdown clearing up homeless session... 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 10 client.4252.objecter successfully canceled tick 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 1 -- 172.21.15.7:0/2745227345 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f620806c8e0 msgr2=0x7f620806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 1 --2- 172.21.15.7:0/2745227345 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f620806c8e0 0x7f620806eda0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f62240b3570 tx=0x7f6218009040 comp rx=0 tx=0).stop 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 10 monclient: shutdown 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 1 -- 172.21.15.7:0/2745227345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6224142710 msgr2=0x7f622413b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 1 --2- 172.21.15.7:0/2745227345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6224142710 0x7f622413b800 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f621002bee0 tx=0x7f62100046c0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 1 -- 172.21.15.7:0/2745227345 shutdown_connections 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 1 --2- 172.21.15.7:0/2745227345 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f620806c8e0 0x7f620806eda0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 1 --2- 172.21.15.7:0/2745227345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6224145900 0x7f6224136a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 1 --2- 172.21.15.7:0/2745227345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6224142710 0x7f622413b800 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.491 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 1 --2- 172.21.15.7:0/2745227345 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f622413fa10 0x7f62241448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.492 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 1 -- 172.21.15.7:0/2745227345 >> 172.21.15.7:0/2745227345 conn(0x7f62240a8a60 msgr2=0x7f6224073df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:58.492 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 1 -- 172.21.15.7:0/2745227345 shutdown_connections 2024-01-21T03:19:58.492 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.504+0000 7f622b0ec640 1 -- 172.21.15.7:0/2745227345 wait complete. 2024-01-21T03:19:58.492 INFO:teuthology.misc.health.smithi007.stderr:2024-01-21T03:19:58.505+0000 7f622b0ec640 1 librados: shutdown 2024-01-21T03:19:58.500 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-21T03:19:58.500 INFO:tasks.ceph:Creating RBD pool 2024-01-21T03:19:58.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-21T03:19:58.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.615+0000 7fde47c75640 10 monclient: get_monmap_and_config 2024-01-21T03:19:58.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.615+0000 7fde47c75640 10 monclient: build_initial_monmap 2024-01-21T03:19:58.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.615+0000 7fde47c75640 10 monclient: monmap: 2024-01-21T03:19:58.601 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:19:58.601 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:58.601 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:19:58.616535+0000 2024-01-21T03:19:58.601 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:19:58.616535+0000 2024-01-21T03:19:58.602 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:58.602 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:58.602 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:19:58.602 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:19:58.602 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:19:58.602 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:19:58.602 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:19:58.602 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:19:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 1 Processor -- start 2024-01-21T03:19:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 1 -- start start 2024-01-21T03:19:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 10 monclient: init 2024-01-21T03:19:58.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-01-21T03:19:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fde400a8b20 0x7fde40145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 10 monclient(hunting): picked mon.noname-c con 0x7fde400a8b20 addr v2:172.21.15.177:3300/0 2024-01-21T03:19:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde400a9d10 0x7fde4013f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 10 monclient(hunting): picked mon.noname-b con 0x7fde400a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde400b2310 addr v1:172.21.15.7:6789/0 2024-01-21T03:19:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde401456f0 con 0x7fde400b2310 2024-01-21T03:19:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde400a98c0 con 0x7fde400a8b20 2024-01-21T03:19:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde401460c0 con 0x7fde400a9d10 2024-01-21T03:19:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.616+0000 7fde47c75640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:58.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.617+0000 7fde47c75640 10 monclient(hunting): authenticate will time out at 1344.932759s 2024-01-21T03:19:58.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.617+0000 7fde45a11640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fde400a8b20 0x7fde40145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:58.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.617+0000 7fde45210640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde400a9d10 0x7fde4013f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:58.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.617+0000 7fde46212640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fde400b2310 0x7fde4013f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59806/0 (socket says 172.21.15.7:59806) 2024-01-21T03:19:58.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.617+0000 7fde46212640 1 -- 172.21.15.7:0/2018632358 learned_addr learned my addr 172.21.15.7:0/2018632358 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:19:58.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.617+0000 7fde45a11640 10 monclient(hunting): get_auth_request con 0x7fde400a8b20 auth_method 0 2024-01-21T03:19:58.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.617+0000 7fde45a11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.617+0000 7fde45a11640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.617+0000 7fde45a11640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.618+0000 7fde45210640 10 monclient(hunting): get_auth_request con 0x7fde400a9d10 auth_method 0 2024-01-21T03:19:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.618+0000 7fde45210640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.618+0000 7fde45210640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.618+0000 7fde45210640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.618+0000 7fde45a11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.618+0000 7fde45a11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.618+0000 7fde45a11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.618+0000 7fde45210640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.618+0000 7fde45210640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.618+0000 7fde45210640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.619+0000 7fde45a11640 10 monclient(hunting): handle_auth_done global_id 4255 payload 274 2024-01-21T03:19:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.619+0000 7fde45a11640 1 -- 172.21.15.7:0/2018632358 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde400a9d10 msgr2=0x7fde4013f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:19:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.619+0000 7fde45a11640 1 --2- 172.21.15.7:0/2018632358 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde400a9d10 0x7fde4013f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.619+0000 7fde45a11640 1 -- 172.21.15.7:0/2018632358 >> v1:172.21.15.7:6789/0 conn(0x7fde400b2310 legacy=0x7fde4013f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.619+0000 7fde45a11640 10 monclient: _finish_hunting 0 2024-01-21T03:19:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.619+0000 7fde45a11640 1 monclient: found mon.noname-c 2024-01-21T03:19:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.619+0000 7fde45a11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.619+0000 7fde45a11640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:19:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.619+0000 7fde45a11640 1 -- 172.21.15.7:0/2018632358 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde40146230 con 0x7fde400a8b20 2024-01-21T03:19:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.621+0000 7fde45a11640 1 --2- 172.21.15.7:0/2018632358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde400a8b20 0x7fde40145b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0097a0 tx=0x7fde3c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9687762ad4a3546 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde44a0f640 1 -- 172.21.15.7:0/2018632358 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c02e070 con 0x7fde400a8b20 2024-01-21T03:19:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde44a0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde44a0f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:19:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde44a0f640 10 monclient: dump: 2024-01-21T03:19:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:19:58.612 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:58.612 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:58.612 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:58.612 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:58.612 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:58.612 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde44a0f640 10 monclient: _finish_auth 0 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde44a0f640 10 monclient: _check_auth_tickets 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde44a0f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde44a0f640 1 -- 172.21.15.7:0/2018632358 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c0315a0 con 0x7fde400a8b20 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde47c75640 5 monclient: authenticate success, global_id 4255 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde47c75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde44a0f640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde47c75640 10 monclient: get_monmap_and_config success 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde47c75640 4 set_mon_vals no callback set 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde44a0f640 1 -- 172.21.15.7:0/2018632358 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c031930 con 0x7fde400a8b20 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde44a0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde44a0f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.625+0000 7fde44a0f640 10 monclient: dump: 2024-01-21T03:19:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:19:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:58.614 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:19:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.626+0000 7fde47c75640 10 monclient: shutdown 2024-01-21T03:19:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.626+0000 7fde47c75640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.626+0000 7fde47c75640 1 -- 172.21.15.7:0/2018632358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde400a8b20 msgr2=0x7fde40145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.626+0000 7fde47c75640 1 --2- 172.21.15.7:0/2018632358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde400a8b20 0x7fde40145b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0097a0 tx=0x7fde3c02b840 comp rx=0 tx=0).stop 2024-01-21T03:19:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.626+0000 7fde47c75640 1 -- 172.21.15.7:0/2018632358 shutdown_connections 2024-01-21T03:19:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.626+0000 7fde47c75640 1 --2- 172.21.15.7:0/2018632358 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde400a9d10 0x7fde4013f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.626+0000 7fde47c75640 1 --2- 172.21.15.7:0/2018632358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde400a8b20 0x7fde40145b80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.626+0000 7fde47c75640 1 -- 172.21.15.7:0/2018632358 >> 172.21.15.7:0/2018632358 conn(0x7fde40066fb0 msgr2=0x7fde400b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.626+0000 7fde47c75640 1 -- 172.21.15.7:0/2018632358 shutdown_connections 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.626+0000 7fde47c75640 1 -- 172.21.15.7:0/2018632358 wait complete. 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.626+0000 7fde47c75640 10 monclient: build_initial_monmap 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.626+0000 7fde47c75640 10 monclient: monmap: 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:19:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 1 librados: starting msgr at 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 1 librados: starting objecter 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 1 Processor -- start 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 1 -- start start 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 1 librados: setting wanted keys 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 1 librados: calling monclient init 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 10 monclient: init 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 10 monclient: _reopen_session rank -1 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde400a8b20 0x7fde40140400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 10 monclient(hunting): picked mon.noname-c con 0x7fde400a8b20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde400a9d10 0x7fde40141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 10 monclient(hunting): picked mon.noname-a con 0x7fde400a9d10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde400b2310 0x7fde401dab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:58.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 10 monclient(hunting): picked mon.noname-b con 0x7fde400b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde40142ad0 con 0x7fde400a9d10 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde401da670 con 0x7fde400b2310 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 10 monclient(hunting): start opening mon connection 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde40139f20 con 0x7fde400a8b20 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 10 monclient(hunting): _renew_subs 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.627+0000 7fde47c75640 10 monclient(hunting): authenticate will time out at 1344.943507s 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde46212640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde400b2310 0x7fde401dab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde45210640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde400a9d10 0x7fde40141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde45a11640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde400a8b20 0x7fde40140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde46212640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde400b2310 0x7fde401dab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36460/0 (socket says 172.21.15.7:36460) 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde46212640 1 -- 172.21.15.7:0/4280136120 learned_addr learned my addr 172.21.15.7:0/4280136120 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde46212640 10 monclient(hunting): get_auth_request con 0x7fde400b2310 auth_method 0 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde46212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde46212640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:58.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde46212640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde45a11640 10 monclient(hunting): get_auth_request con 0x7fde400a8b20 auth_method 0 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde45a11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde45a11640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde45a11640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde45210640 10 monclient(hunting): get_auth_request con 0x7fde400a9d10 auth_method 0 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde45210640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde45210640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde45210640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde46212640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde46212640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.628+0000 7fde46212640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.629+0000 7fde45a11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.629+0000 7fde45a11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.629+0000 7fde45a11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.629+0000 7fde45210640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:19:58.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.629+0000 7fde45210640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:19:58.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.629+0000 7fde45210640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:19:58.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.631+0000 7fde46212640 10 monclient(hunting): handle_auth_done global_id 4258 payload 824 2024-01-21T03:19:58.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.631+0000 7fde46212640 1 -- 172.21.15.7:0/4280136120 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde400a8b20 msgr2=0x7fde40140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:58.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.631+0000 7fde46212640 1 --2- 172.21.15.7:0/4280136120 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde400a8b20 0x7fde40140400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.631+0000 7fde46212640 1 -- 172.21.15.7:0/4280136120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde400a9d10 msgr2=0x7fde40141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:58.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.631+0000 7fde46212640 1 --2- 172.21.15.7:0/4280136120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde400a9d10 0x7fde40141210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:58.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.631+0000 7fde46212640 10 monclient: _finish_hunting 0 2024-01-21T03:19:58.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.631+0000 7fde46212640 1 monclient: found mon.noname-b 2024-01-21T03:19:58.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.631+0000 7fde46212640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:19:58.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.631+0000 7fde46212640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:19:58.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.631+0000 7fde46212640 1 -- 172.21.15.7:0/4280136120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3c009400 con 0x7fde400b2310 2024-01-21T03:19:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.633+0000 7fde46212640 1 --2- 172.21.15.7:0/4280136120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde400b2310 0x7fde401dab90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fde3800caa0 tx=0x7fde3800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.633+0000 7fde367fc640 10 client.?.objecter ms_handle_connect 0x7fde400b2310 2024-01-21T03:19:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.633+0000 7fde367fc640 10 client.?.objecter resend_mon_ops 2024-01-21T03:19:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde367fc640 1 -- 172.21.15.7:0/4280136120 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3800ac00 con 0x7fde400b2310 2024-01-21T03:19:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:19:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde367fc640 10 monclient: dump: 2024-01-21T03:19:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:19:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde367fc640 10 monclient: _finish_auth 0 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde367fc640 10 monclient: _check_auth_tickets 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde367fc640 1 -- 172.21.15.7:0/4280136120 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3800ad60 con 0x7fde400b2310 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde47c75640 5 monclient: authenticate success, global_id 4258 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde47c75640 10 monclient: _renew_subs 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde47c75640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde47c75640 1 -- 172.21.15.7:0/4280136120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde40141750 con 0x7fde400b2310 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde367fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde367fc640 1 -- 172.21.15.7:0/4280136120 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38012600 con 0x7fde400b2310 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:19:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:19:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde367fc640 10 monclient: dump: 2024-01-21T03:19:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:19:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:19:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:19:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:19:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:19:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:19:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:19:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:19:58.627 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:19:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde37fff640 4 set_mon_vals no callback set 2024-01-21T03:19:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde47c75640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:19:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde47c75640 10 monclient: _renew_subs 2024-01-21T03:19:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde47c75640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde47c75640 1 -- 172.21.15.7:0/4280136120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde401db2b0 con 0x7fde400b2310 2024-01-21T03:19:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.637+0000 7fde47c75640 1 librados: init done 2024-01-21T03:19:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.638+0000 7fde47c75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.638+0000 7fde47c75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.638+0000 7fde47c75640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:58.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.638+0000 7fde47c75640 1 -- 172.21.15.7:0/4280136120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4004e970 con 0x7fde400b2310 2024-01-21T03:19:58.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.880+0000 7fde367fc640 1 -- 172.21.15.7:0/4280136120 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fde38007070 con 0x7fde400b2310 2024-01-21T03:19:58.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.881+0000 7fde367fc640 1 --2- 172.21.15.7:0/4280136120 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fde2406c860 0x7fde2406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:19:58.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.881+0000 7fde45a11640 1 --2- 172.21.15.7:0/4280136120 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fde2406c860 0x7fde2406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:19:58.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.881+0000 7fde45a11640 10 monclient: get_auth_request con 0x7fde2406c860 auth_method 0 2024-01-21T03:19:58.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.881+0000 7fde367fc640 10 client.4258.objecter ms_handle_connect 0x7fde2406c860 2024-01-21T03:19:58.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.881+0000 7fde45a11640 1 --2- 172.21.15.7:0/4280136120 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fde2406c860 0x7fde2406ed20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fde3c036000 tx=0x7fde3c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:19:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.883+0000 7fde367fc640 1 -- 172.21.15.7:0/4280136120 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fde38058570 con 0x7fde400b2310 2024-01-21T03:19:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.883+0000 7fde367fc640 10 client.4258.objecter ms_dispatch 0x7fde40000b90 osd_map(23..23 src has 1..23) v4 2024-01-21T03:19:58.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.883+0000 7fde367fc640 3 client.4258.objecter handle_osd_map got epochs [23,23] > 0 2024-01-21T03:19:58.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.883+0000 7fde367fc640 3 client.4258.objecter handle_osd_map decoding full epoch 23 2024-01-21T03:19:58.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.883+0000 7fde367fc640 20 client.4258.objecter dump_active .. 0 homeless 2024-01-21T03:19:58.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.895+0000 7fde367fc640 1 -- 172.21.15.7:0/4280136120 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fde38087e60 con 0x7fde400b2310 2024-01-21T03:19:58.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.895+0000 7fde367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:19:58.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:58.895+0000 7fde367fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:19:59.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:59.098+0000 7fde47c75640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-01-21T03:19:59.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:59.098+0000 7fde47c75640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-01-21T03:19:59.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:59.098+0000 7fde47c75640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:19:59.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:19:59.098+0000 7fde47c75640 1 -- 172.21.15.7:0/4280136120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fde401db560 con 0x7fde400b2310 2024-01-21T03:19:59.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.005+0000 7fde367fc640 1 -- 172.21.15.7:0/4280136120 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v24) v1 ==== 109+0+0 (secure 0 0 0) 0x7fde380210a0 con 0x7fde400b2310 2024-01-21T03:19:59.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.005+0000 7fde367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-01-21T03:19:59.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.005+0000 7fde367fc640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-01-21T03:19:59.991 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'rbd' created 2024-01-21T03:19:59.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 10 librados: watch_flush enter 2024-01-21T03:19:59.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 10 librados: watch_flush exit 2024-01-21T03:19:59.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 20 client.4258.objecter shutdown clearing up homeless session... 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 10 client.4258.objecter successfully canceled tick 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 1 -- 172.21.15.7:0/4280136120 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fde2406c860 msgr2=0x7fde2406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 1 --2- 172.21.15.7:0/4280136120 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fde2406c860 0x7fde2406ed20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fde3c036000 tx=0x7fde3c03d040 comp rx=0 tx=0).stop 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 10 monclient: shutdown 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 1 -- 172.21.15.7:0/4280136120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde400b2310 msgr2=0x7fde401dab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 1 --2- 172.21.15.7:0/4280136120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde400b2310 0x7fde401dab90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fde3800caa0 tx=0x7fde3800cf60 comp rx=0 tx=0).stop 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 1 -- 172.21.15.7:0/4280136120 shutdown_connections 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 1 --2- 172.21.15.7:0/4280136120 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fde2406c860 0x7fde2406ed20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 1 --2- 172.21.15.7:0/4280136120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde400b2310 0x7fde401dab90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 1 --2- 172.21.15.7:0/4280136120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde400a9d10 0x7fde40141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 1 --2- 172.21.15.7:0/4280136120 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde400a8b20 0x7fde40140400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 1 -- 172.21.15.7:0/4280136120 >> 172.21.15.7:0/4280136120 conn(0x7fde40066fb0 msgr2=0x7fde400b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 1 -- 172.21.15.7:0/4280136120 shutdown_connections 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 1 -- 172.21.15.7:0/4280136120 wait complete. 2024-01-21T03:19:59.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.007+0000 7fde47c75640 1 librados: shutdown 2024-01-21T03:20:00.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-21T03:20:00.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.125+0000 7f5b88db0640 10 monclient: get_monmap_and_config 2024-01-21T03:20:00.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.125+0000 7f5b88db0640 10 monclient: build_initial_monmap 2024-01-21T03:20:00.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.125+0000 7f5b88db0640 10 monclient: monmap: 2024-01-21T03:20:00.111 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:00.111 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:00.111 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:00.126005+0000 2024-01-21T03:20:00.111 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:00.126005+0000 2024-01-21T03:20:00.111 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:00.112 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.125+0000 7f5b88db0640 1 Processor -- start 2024-01-21T03:20:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.125+0000 7f5b88db0640 1 -- start start 2024-01-21T03:20:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.125+0000 7f5b88db0640 10 monclient: init 2024-01-21T03:20:00.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:00.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-01-21T03:20:00.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5b840a8a40 0x7f5b84145b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:00.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 10 monclient(hunting): picked mon.noname-a con 0x7f5b840a8a40 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:00.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b840b2210 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:00.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5b840a8e20 0x7f5b8414c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 10 monclient(hunting): picked mon.noname-b con 0x7f5b840a8e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b84145700 con 0x7f5b840b2210 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b840aa7b0 con 0x7f5b840a8a40 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b840aa9b0 con 0x7f5b840a8e20 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.126+0000 7f5b88db0640 10 monclient(hunting): authenticate will time out at 1346.442232s 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.127+0000 7f5b79d9b640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f5b840b2210 0x7f5b8413f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:44314/0 (socket says 172.21.15.7:44314) 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.127+0000 7f5b79d9b640 1 -- 172.21.15.7:0/4021204321 learned_addr learned my addr 172.21.15.7:0/4021204321 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.128+0000 7f5b82d9d640 1 --2- 172.21.15.7:0/4021204321 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5b840a8e20 0x7f5b8414c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.128+0000 7f5b82d9d640 10 monclient(hunting): get_auth_request con 0x7f5b840a8e20 auth_method 0 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.128+0000 7f5b82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:00.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.128+0000 7f5b82d9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:00.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.128+0000 7f5b82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:00.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.128+0000 7f5b8259c640 1 --2- 172.21.15.7:0/4021204321 >> v2:172.21.15.7:3300/0 conn(0x7f5b840a8a40 0x7f5b84145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:00.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.128+0000 7f5b82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:00.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.128+0000 7f5b82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:00.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.128+0000 7f5b82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:00.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.129+0000 7f5b8259c640 10 monclient(hunting): get_auth_request con 0x7f5b840a8a40 auth_method 0 2024-01-21T03:20:00.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.129+0000 7f5b8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:00.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.129+0000 7f5b8259c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:00.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.129+0000 7f5b8259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:00.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.129+0000 7f5b8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:00.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.129+0000 7f5b8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:00.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.129+0000 7f5b8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:00.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.130+0000 7f5b81d9b640 1 -- 172.21.15.7:0/4021204321 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192637194 0 0) 0x7f5b84145700 con 0x7f5b840b2210 2024-01-21T03:20:00.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.130+0000 7f5b81d9b640 10 monclient(hunting): _negotiate 2024-01-21T03:20:00.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.130+0000 7f5b81d9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:00.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.130+0000 7f5b81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:00.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.130+0000 7f5b81d9b640 10 monclient(hunting): my global_id is 4261 2024-01-21T03:20:00.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.130+0000 7f5b81d9b640 1 -- 172.21.15.7:0/4021204321 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b70003520 con 0x7f5b840b2210 2024-01-21T03:20:00.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.130+0000 7f5b82d9d640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2024-01-21T03:20:00.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.130+0000 7f5b82d9d640 1 -- 172.21.15.7:0/4021204321 >> v2:172.21.15.7:3300/0 conn(0x7f5b840a8a40 msgr2=0x7f5b84145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:00.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.130+0000 7f5b82d9d640 1 --2- 172.21.15.7:0/4021204321 >> v2:172.21.15.7:3300/0 conn(0x7f5b840a8a40 0x7f5b84145b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:00.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.130+0000 7f5b82d9d640 1 -- 172.21.15.7:0/4021204321 >> v1:172.21.15.177:6789/0 conn(0x7f5b840b2210 legacy=0x7f5b8413f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:00.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.130+0000 7f5b82d9d640 10 monclient: _finish_hunting 0 2024-01-21T03:20:00.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.130+0000 7f5b82d9d640 1 monclient: found mon.noname-b 2024-01-21T03:20:00.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.131+0000 7f5b82d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:00.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.131+0000 7f5b82d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:00.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.131+0000 7f5b82d9d640 1 -- 172.21.15.7:0/4021204321 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b840aac90 con 0x7f5b840a8e20 2024-01-21T03:20:00.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.133+0000 7f5b82d9d640 1 --2- 172.21.15.7:0/4021204321 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5b840a8e20 0x7f5b8414c8d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009d90 tx=0x7f5b7c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5738e0f6d190cc77 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:00.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b81d9b640 1 -- 172.21.15.7:0/4021204321 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c02cc50 con 0x7f5b840a8e20 2024-01-21T03:20:00.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b81d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:00.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b81d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:00.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b81d9b640 10 monclient: dump: 2024-01-21T03:20:00.125 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:00.125 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b81d9b640 10 monclient: _finish_auth 0 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b81d9b640 10 monclient: _check_auth_tickets 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b81d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b81d9b640 1 -- 172.21.15.7:0/4021204321 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c02cdb0 con 0x7f5b840a8e20 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b81d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b81d9b640 1 -- 172.21.15.7:0/4021204321 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c034660 con 0x7f5b840a8e20 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b81d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:00.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b81d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b81d9b640 10 monclient: dump: 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b88db0640 5 monclient: authenticate success, global_id 4235 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b88db0640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.139+0000 7f5b88db0640 4 set_mon_vals no callback set 2024-01-21T03:20:00.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.140+0000 7f5b88db0640 10 monclient: shutdown 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.140+0000 7f5b88db0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.140+0000 7f5b88db0640 1 -- 172.21.15.7:0/4021204321 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5b840a8e20 msgr2=0x7f5b8414c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.140+0000 7f5b88db0640 1 --2- 172.21.15.7:0/4021204321 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5b840a8e20 0x7f5b8414c8d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009d90 tx=0x7f5b7c02bea0 comp rx=0 tx=0).stop 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.140+0000 7f5b88db0640 1 -- 172.21.15.7:0/4021204321 shutdown_connections 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.140+0000 7f5b88db0640 1 --2- 172.21.15.7:0/4021204321 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5b840a8e20 0x7f5b8414c8d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.140+0000 7f5b88db0640 1 --2- 172.21.15.7:0/4021204321 >> v2:172.21.15.7:3300/0 conn(0x7f5b840a8a40 0x7f5b84145b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.140+0000 7f5b88db0640 1 -- 172.21.15.7:0/4021204321 >> 172.21.15.7:0/4021204321 conn(0x7f5b84066f10 msgr2=0x7f5b840b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.140+0000 7f5b88db0640 1 -- 172.21.15.7:0/4021204321 shutdown_connections 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.140+0000 7f5b88db0640 1 -- 172.21.15.7:0/4021204321 wait complete. 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.141+0000 7f5b88db0640 10 monclient: build_initial_monmap 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.141+0000 7f5b88db0640 10 monclient: monmap: 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:00.131 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.141+0000 7f5b88db0640 1 librados: starting msgr at 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.141+0000 7f5b88db0640 1 librados: starting objecter 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 1 Processor -- start 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 1 -- start start 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 1 librados: setting wanted keys 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 1 librados: calling monclient init 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 10 monclient: init 2024-01-21T03:20:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b840a8a40 0x7f5b84140be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 10 monclient(hunting): picked mon.noname-a con 0x7f5b840a8a40 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5b840a8e20 0x7f5b84143780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 10 monclient(hunting): picked mon.noname-c con 0x7f5b840a8e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b840b2210 0x7f5b84143cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 10 monclient(hunting): picked mon.noname-b con 0x7f5b840b2210 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b84141a60 con 0x7f5b840a8a40 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b84144200 con 0x7f5b840b2210 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8413fde0 con 0x7f5b840a8e20 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.142+0000 7f5b88db0640 10 monclient(hunting): authenticate will time out at 1346.458428s 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b79d9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5b840a8e20 0x7f5b84143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b8259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b840a8a40 0x7f5b84140be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b82d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b840b2210 0x7f5b84143cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b82d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b840b2210 0x7f5b84143cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36468/0 (socket says 172.21.15.7:36468) 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b82d9d640 1 -- 172.21.15.7:0/1055659288 learned_addr learned my addr 172.21.15.7:0/1055659288 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b82d9d640 10 monclient(hunting): get_auth_request con 0x7f5b840b2210 auth_method 0 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b82d9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b79d9b640 10 monclient(hunting): get_auth_request con 0x7f5b840a8e20 auth_method 0 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b79d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b79d9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b8259c640 10 monclient(hunting): get_auth_request con 0x7f5b840a8a40 auth_method 0 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b8259c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b8259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.143+0000 7f5b82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.144+0000 7f5b79d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.144+0000 7f5b79d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.144+0000 7f5b79d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.144+0000 7f5b8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.144+0000 7f5b8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.144+0000 7f5b8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.146+0000 7f5b82d9d640 10 monclient(hunting): handle_auth_done global_id 4264 payload 824 2024-01-21T03:20:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.146+0000 7f5b82d9d640 1 -- 172.21.15.7:0/1055659288 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5b840a8e20 msgr2=0x7f5b84143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.146+0000 7f5b82d9d640 1 --2- 172.21.15.7:0/1055659288 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5b840a8e20 0x7f5b84143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.146+0000 7f5b82d9d640 1 -- 172.21.15.7:0/1055659288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b840a8a40 msgr2=0x7f5b84140be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.146+0000 7f5b82d9d640 1 --2- 172.21.15.7:0/1055659288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b840a8a40 0x7f5b84140be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.146+0000 7f5b82d9d640 10 monclient: _finish_hunting 0 2024-01-21T03:20:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.146+0000 7f5b82d9d640 1 monclient: found mon.noname-b 2024-01-21T03:20:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.146+0000 7f5b82d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.146+0000 7f5b82d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.146+0000 7f5b82d9d640 1 -- 172.21.15.7:0/1055659288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c009a70 con 0x7f5b840b2210 2024-01-21T03:20:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.148+0000 7f5b82d9d640 1 --2- 172.21.15.7:0/1055659288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b840b2210 0x7f5b84143cc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009d90 tx=0x7f5b7c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:00.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.148+0000 7f5b7b7fe640 10 client.?.objecter ms_handle_connect 0x7f5b840b2210 2024-01-21T03:20:00.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.148+0000 7f5b7b7fe640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:00.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b7b7fe640 1 -- 172.21.15.7:0/1055659288 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c040650 con 0x7f5b840b2210 2024-01-21T03:20:00.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b7b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b7b7fe640 10 monclient: dump: 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b7b7fe640 10 monclient: _finish_auth 0 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b7b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:20:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b7b7fe640 1 -- 172.21.15.7:0/1055659288 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c0407b0 con 0x7f5b840b2210 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b88db0640 5 monclient: authenticate success, global_id 4264 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b88db0640 10 monclient: _renew_subs 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b88db0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b88db0640 1 -- 172.21.15.7:0/1055659288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8413fc50 con 0x7f5b840b2210 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b7b7fe640 1 -- 172.21.15.7:0/1055659288 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c040b60 con 0x7f5b840b2210 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b7b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b7b7fe640 10 monclient: dump: 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:00.141 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b8159a640 4 set_mon_vals no callback set 2024-01-21T03:20:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b88db0640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b88db0640 10 monclient: _renew_subs 2024-01-21T03:20:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b88db0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b88db0640 1 -- 172.21.15.7:0/1055659288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b84141120 con 0x7f5b840b2210 2024-01-21T03:20:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.152+0000 7f5b88db0640 1 librados: init done 2024-01-21T03:20:00.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.153+0000 7f5b88db0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:00.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.153+0000 7f5b88db0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:00.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.153+0000 7f5b88db0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:00.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.153+0000 7f5b88db0640 1 -- 172.21.15.7:0/1055659288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b840aa1c0 con 0x7f5b840b2210 2024-01-21T03:20:00.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.172+0000 7f5b7b7fe640 1 -- 172.21.15.7:0/1055659288 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5b7c040cc0 con 0x7f5b840b2210 2024-01-21T03:20:00.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.172+0000 7f5b7b7fe640 1 --2- 172.21.15.7:0/1055659288 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f5b6406c860 0x7f5b6406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:00.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.172+0000 7f5b8259c640 1 --2- 172.21.15.7:0/1055659288 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f5b6406c860 0x7f5b6406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:00.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.172+0000 7f5b8259c640 10 monclient: get_auth_request con 0x7f5b6406c860 auth_method 0 2024-01-21T03:20:00.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.173+0000 7f5b8259c640 1 --2- 172.21.15.7:0/1055659288 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f5b6406c860 0x7f5b6406ed20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009b40 tx=0x7f5b74009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:00.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.173+0000 7f5b7b7fe640 10 client.4264.objecter ms_handle_connect 0x7f5b6406c860 2024-01-21T03:20:00.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.402+0000 7f5b7b7fe640 1 -- 172.21.15.7:0/1055659288 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f5b7c07b260 con 0x7f5b840b2210 2024-01-21T03:20:00.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.402+0000 7f5b7b7fe640 10 client.4264.objecter ms_dispatch 0x7f5b84000b90 osd_map(24..24 src has 1..24) v4 2024-01-21T03:20:00.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.402+0000 7f5b7b7fe640 3 client.4264.objecter handle_osd_map got epochs [24,24] > 0 2024-01-21T03:20:00.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.402+0000 7f5b7b7fe640 3 client.4264.objecter handle_osd_map decoding full epoch 24 2024-01-21T03:20:00.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.402+0000 7f5b7b7fe640 20 client.4264.objecter dump_active .. 0 homeless 2024-01-21T03:20:00.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.416+0000 7f5b7b7fe640 1 -- 172.21.15.7:0/1055659288 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5b7c0775a0 con 0x7f5b840b2210 2024-01-21T03:20:00.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.416+0000 7f5b7b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:00.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.416+0000 7f5b7b7fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:00.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.604+0000 7f5b88db0640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-01-21T03:20:00.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.604+0000 7f5b88db0640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-01-21T03:20:00.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.604+0000 7f5b88db0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:00.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:00.604+0000 7f5b88db0640 1 -- 172.21.15.7:0/1055659288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5b84141c10 con 0x7f5b840b2210 2024-01-21T03:20:01.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.584+0000 7f5b7b7fe640 1 -- 172.21.15.7:0/1055659288 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v25) v1 ==== 173+0+0 (secure 0 0 0) 0x7f5b84141c10 con 0x7f5b840b2210 2024-01-21T03:20:01.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.584+0000 7f5b7b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-01-21T03:20:01.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.584+0000 7f5b7b7fe640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-01-21T03:20:01.570 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-21T03:20:01.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.585+0000 7f5b88db0640 10 librados: watch_flush enter 2024-01-21T03:20:01.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.585+0000 7f5b88db0640 10 librados: watch_flush exit 2024-01-21T03:20:01.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.585+0000 7f5b88db0640 20 client.4264.objecter shutdown clearing up homeless session... 2024-01-21T03:20:01.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.585+0000 7f5b88db0640 10 client.4264.objecter successfully canceled tick 2024-01-21T03:20:01.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 1 -- 172.21.15.7:0/1055659288 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f5b6406c860 msgr2=0x7f5b6406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:01.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 1 --2- 172.21.15.7:0/1055659288 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f5b6406c860 0x7f5b6406ed20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009b40 tx=0x7f5b74009190 comp rx=0 tx=0).stop 2024-01-21T03:20:01.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 10 monclient: shutdown 2024-01-21T03:20:01.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:01.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 1 -- 172.21.15.7:0/1055659288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b840b2210 msgr2=0x7f5b84143cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:01.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 1 --2- 172.21.15.7:0/1055659288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b840b2210 0x7f5b84143cc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009d90 tx=0x7f5b7c004a40 comp rx=0 tx=0).stop 2024-01-21T03:20:01.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 1 -- 172.21.15.7:0/1055659288 shutdown_connections 2024-01-21T03:20:01.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 1 --2- 172.21.15.7:0/1055659288 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f5b6406c860 0x7f5b6406ed20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:01.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 1 --2- 172.21.15.7:0/1055659288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b840b2210 0x7f5b84143cc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:01.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 1 --2- 172.21.15.7:0/1055659288 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5b840a8e20 0x7f5b84143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:01.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 1 --2- 172.21.15.7:0/1055659288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b840a8a40 0x7f5b84140be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:01.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 1 -- 172.21.15.7:0/1055659288 >> 172.21.15.7:0/1055659288 conn(0x7f5b84066f10 msgr2=0x7f5b8413a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:01.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 1 -- 172.21.15.7:0/1055659288 shutdown_connections 2024-01-21T03:20:01.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 1 -- 172.21.15.7:0/1055659288 wait complete. 2024-01-21T03:20:01.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.586+0000 7f5b88db0640 1 librados: shutdown 2024-01-21T03:20:01.583 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-21T03:20:01.583 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-21T03:20:01.583 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-21T03:20:01.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-21T03:20:01.584 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-21T03:20:01.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.705+0000 7f46f6ab4640 10 monclient: get_monmap_and_config 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.705+0000 7f46f6ab4640 10 monclient: build_initial_monmap 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.705+0000 7f46f6ab4640 10 monclient: monmap: 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:01.706006+0000 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:01.706006+0000 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:01.691 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:01.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.705+0000 7f46f6ab4640 1 Processor -- start 2024-01-21T03:20:01.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.705+0000 7f46f6ab4640 1 -- start start 2024-01-21T03:20:01.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.705+0000 7f46f6ab4640 10 monclient: init 2024-01-21T03:20:01.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:01.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-01-21T03:20:01.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46f00a8a40 0x7f46f0145b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:01.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 10 monclient(hunting): picked mon.noname-b con 0x7f46f00a8a40 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:01.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46f01460a0 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:01.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f46f013f8b0 0x7f46f014c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:01.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 10 monclient(hunting): picked mon.noname-a con 0x7f46f013f8b0 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f00aa9b0 con 0x7f46f01460a0 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f00aa7d0 con 0x7f46f013f8b0 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f00aac90 con 0x7f46f00a8a40 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.706+0000 7f46f6ab4640 10 monclient(hunting): authenticate will time out at 1348.022352s 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.707+0000 7f46f5051640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f46f013f8b0 0x7f46f014c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.707+0000 7f46effff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f46f01460a0 0x7f46f013f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:44330/0 (socket says 172.21.15.7:44330) 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.707+0000 7f46effff640 1 -- 172.21.15.7:0/1506503398 learned_addr learned my addr 172.21.15.7:0/1506503398 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.707+0000 7f46f4850640 1 --2- 172.21.15.7:0/1506503398 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46f00a8a40 0x7f46f0145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.707+0000 7f46f4850640 10 monclient(hunting): get_auth_request con 0x7f46f00a8a40 auth_method 0 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.707+0000 7f46f4850640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.707+0000 7f46f4850640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.707+0000 7f46f4850640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:01.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.707+0000 7f46f5051640 10 monclient(hunting): get_auth_request con 0x7f46f013f8b0 auth_method 0 2024-01-21T03:20:01.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.707+0000 7f46f5051640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:01.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.707+0000 7f46f5051640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:01.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.707+0000 7f46f5051640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:01.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.708+0000 7f46f4850640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:01.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.708+0000 7f46f4850640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:01.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.708+0000 7f46f4850640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:01.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.708+0000 7f46f5051640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:01.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.708+0000 7f46f5051640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:01.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.708+0000 7f46f5051640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:01.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.710+0000 7f46f4850640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-01-21T03:20:01.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.710+0000 7f46f4850640 1 -- 172.21.15.7:0/1506503398 >> v2:172.21.15.7:3300/0 conn(0x7f46f013f8b0 msgr2=0x7f46f014c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:01.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.710+0000 7f46f4850640 1 --2- 172.21.15.7:0/1506503398 >> v2:172.21.15.7:3300/0 conn(0x7f46f013f8b0 0x7f46f014c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:01.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.710+0000 7f46f4850640 1 -- 172.21.15.7:0/1506503398 >> v1:172.21.15.177:6789/0 conn(0x7f46f01460a0 legacy=0x7f46f013f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:01.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.710+0000 7f46f4850640 10 monclient: _finish_hunting 0 2024-01-21T03:20:01.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.710+0000 7f46f4850640 1 monclient: found mon.noname-b 2024-01-21T03:20:01.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.710+0000 7f46f4850640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:01.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.710+0000 7f46f4850640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:01.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.710+0000 7f46f4850640 1 -- 172.21.15.7:0/1506503398 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f00aaf70 con 0x7f46f00a8a40 2024-01-21T03:20:01.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.712+0000 7f46f4850640 1 --2- 172.21.15.7:0/1506503398 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46f00a8a40 0x7f46f0145b60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f46e40097a0 tx=0x7f46e402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e294cfb9a55f7e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:01.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46ef7fe640 1 -- 172.21.15.7:0/1506503398 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4030800 con 0x7f46f00a8a40 2024-01-21T03:20:01.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:01.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:01.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46ef7fe640 10 monclient: dump: 2024-01-21T03:20:01.702 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46ef7fe640 10 monclient: _finish_auth 0 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46ef7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46ef7fe640 1 -- 172.21.15.7:0/1506503398 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e4030960 con 0x7f46f00a8a40 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:01.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46ef7fe640 1 -- 172.21.15.7:0/1506503398 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4030cf0 con 0x7f46f00a8a40 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46ef7fe640 10 monclient: dump: 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46f6ab4640 5 monclient: authenticate success, global_id 4241 2024-01-21T03:20:01.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46f6ab4640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:01.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.716+0000 7f46f6ab4640 4 set_mon_vals no callback set 2024-01-21T03:20:01.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.717+0000 7f46f6ab4640 10 monclient: shutdown 2024-01-21T03:20:01.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.717+0000 7f46f6ab4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.717+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1506503398 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46f00a8a40 msgr2=0x7f46f0145b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.717+0000 7f46f6ab4640 1 --2- 172.21.15.7:0/1506503398 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46f00a8a40 0x7f46f0145b60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f46e40097a0 tx=0x7f46e402b940 comp rx=0 tx=0).stop 2024-01-21T03:20:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.717+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1506503398 shutdown_connections 2024-01-21T03:20:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.717+0000 7f46f6ab4640 1 --2- 172.21.15.7:0/1506503398 >> v2:172.21.15.7:3300/0 conn(0x7f46f013f8b0 0x7f46f014c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.717+0000 7f46f6ab4640 1 --2- 172.21.15.7:0/1506503398 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46f00a8a40 0x7f46f0145b60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.717+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1506503398 >> 172.21.15.7:0/1506503398 conn(0x7f46f0066f50 msgr2=0x7f46f00b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.717+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1506503398 shutdown_connections 2024-01-21T03:20:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.717+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1506503398 wait complete. 2024-01-21T03:20:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.718+0000 7f46f6ab4640 10 monclient: build_initial_monmap 2024-01-21T03:20:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.718+0000 7f46f6ab4640 10 monclient: monmap: 2024-01-21T03:20:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:01.708 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.718+0000 7f46f6ab4640 1 librados: starting msgr at 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.718+0000 7f46f6ab4640 1 librados: starting objecter 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 1 Processor -- start 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 1 -- start start 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 1 librados: setting wanted keys 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 1 librados: calling monclient init 2024-01-21T03:20:01.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 10 monclient: init 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f00a8a40 0x7f46f0144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 10 monclient(hunting): picked mon.noname-b con 0x7f46f00a8a40 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f013f8b0 0x7f46f0142030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 10 monclient(hunting): picked mon.noname-a con 0x7f46f013f8b0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46f01460a0 0x7f46f01d6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 10 monclient(hunting): picked mon.noname-c con 0x7f46f01460a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f01429d0 con 0x7f46f013f8b0 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f0141d00 con 0x7f46f00a8a40 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f0142570 con 0x7f46f01460a0 2024-01-21T03:20:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.719+0000 7f46f6ab4640 10 monclient(hunting): authenticate will time out at 1348.035656s 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.720+0000 7f46f5051640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46f01460a0 0x7f46f01d6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.720+0000 7f46effff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f013f8b0 0x7f46f0142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.720+0000 7f46f4850640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f00a8a40 0x7f46f0144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.720+0000 7f46f5051640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46f01460a0 0x7f46f01d6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45736/0 (socket says 172.21.15.7:45736) 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.720+0000 7f46f5051640 1 -- 172.21.15.7:0/1311754282 learned_addr learned my addr 172.21.15.7:0/1311754282 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.720+0000 7f46f5051640 10 monclient(hunting): get_auth_request con 0x7f46f01460a0 auth_method 0 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.720+0000 7f46f5051640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.720+0000 7f46f5051640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.720+0000 7f46f5051640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.720+0000 7f46effff640 10 monclient(hunting): get_auth_request con 0x7f46f013f8b0 auth_method 0 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.720+0000 7f46effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.720+0000 7f46effff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.720+0000 7f46effff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.721+0000 7f46f4850640 10 monclient(hunting): get_auth_request con 0x7f46f00a8a40 auth_method 0 2024-01-21T03:20:01.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.721+0000 7f46f4850640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:01.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.721+0000 7f46f4850640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:01.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.721+0000 7f46f4850640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:01.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.721+0000 7f46f5051640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:01.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.721+0000 7f46f5051640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:01.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.721+0000 7f46f5051640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:01.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.721+0000 7f46effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:01.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.721+0000 7f46effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:01.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.721+0000 7f46effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.721+0000 7f46f4850640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.721+0000 7f46f4850640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.721+0000 7f46f4850640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.725+0000 7f46f5051640 10 monclient(hunting): handle_auth_done global_id 4244 payload 824 2024-01-21T03:20:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.725+0000 7f46f5051640 1 -- 172.21.15.7:0/1311754282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f00a8a40 msgr2=0x7f46f0144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.725+0000 7f46f5051640 1 --2- 172.21.15.7:0/1311754282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f00a8a40 0x7f46f0144c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.725+0000 7f46f5051640 1 -- 172.21.15.7:0/1311754282 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f013f8b0 msgr2=0x7f46f0142030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.725+0000 7f46f5051640 1 --2- 172.21.15.7:0/1311754282 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f013f8b0 0x7f46f0142030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.725+0000 7f46f5051640 10 monclient: _finish_hunting 0 2024-01-21T03:20:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.725+0000 7f46f5051640 1 monclient: found mon.noname-c 2024-01-21T03:20:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.725+0000 7f46f5051640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:01.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.725+0000 7f46f5051640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:20:01.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.725+0000 7f46f5051640 1 -- 172.21.15.7:0/1311754282 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e4009400 con 0x7f46f01460a0 2024-01-21T03:20:01.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.728+0000 7f46f5051640 1 --2- 172.21.15.7:0/1311754282 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46f01460a0 0x7f46f01d6ee0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f46e800efc0 tx=0x7f46e800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:01.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.728+0000 7f46ed7fa640 10 client.?.objecter ms_handle_connect 0x7f46f01460a0 2024-01-21T03:20:01.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.728+0000 7f46ed7fa640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46ed7fa640 1 -- 172.21.15.7:0/1311754282 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e8009ca0 con 0x7f46f01460a0 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46ed7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46ed7fa640 10 monclient: dump: 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:01.717 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:01.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46ed7fa640 10 monclient: _finish_auth 0 2024-01-21T03:20:01.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46ed7fa640 10 monclient: _check_auth_tickets 2024-01-21T03:20:01.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:01.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46f6ab4640 5 monclient: authenticate success, global_id 4244 2024-01-21T03:20:01.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46f6ab4640 10 monclient: _renew_subs 2024-01-21T03:20:01.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46f6ab4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:01.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1311754282 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f01d7420 con 0x7f46f01460a0 2024-01-21T03:20:01.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46f6ab4640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:01.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46f6ab4640 10 monclient: _renew_subs 2024-01-21T03:20:01.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46f6ab4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:01.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1311754282 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f01d7930 con 0x7f46f01460a0 2024-01-21T03:20:01.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.731+0000 7f46f6ab4640 1 librados: init done 2024-01-21T03:20:01.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.732+0000 7f46f6ab4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:01.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.732+0000 7f46f6ab4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:01.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.732+0000 7f46f6ab4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:01.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.732+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1311754282 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f0142b80 con 0x7f46f01460a0 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.733+0000 7f46ed7fa640 1 -- 172.21.15.7:0/1311754282 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e800f040 con 0x7f46f01460a0 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.733+0000 7f46ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.733+0000 7f46eeffd640 4 set_mon_vals no callback set 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.734+0000 7f46ed7fa640 1 -- 172.21.15.7:0/1311754282 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e80146a0 con 0x7f46f01460a0 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.734+0000 7f46ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.734+0000 7f46ed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:01.734+0000 7f46ed7fa640 10 monclient: dump: 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:01.720 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:01.721 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:01.721 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.007+0000 7f46ed7fa640 1 -- 172.21.15.7:0/1311754282 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f46e8014800 con 0x7f46f01460a0 2024-01-21T03:20:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.007+0000 7f46ed7fa640 1 --2- 172.21.15.7:0/1311754282 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f46d406c860 0x7f46d406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.008+0000 7f46f4850640 1 --2- 172.21.15.7:0/1311754282 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f46d406c860 0x7f46d406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.008+0000 7f46f4850640 10 monclient: get_auth_request con 0x7f46d406c860 auth_method 0 2024-01-21T03:20:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.008+0000 7f46ed7fa640 10 client.4244.objecter ms_handle_connect 0x7f46d406c860 2024-01-21T03:20:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.008+0000 7f46f4850640 1 --2- 172.21.15.7:0/1311754282 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f46d406c860 0x7f46d406ed20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f46e4036000 tx=0x7f46e4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:02.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.021+0000 7f46ed7fa640 1 -- 172.21.15.7:0/1311754282 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f46e8059420 con 0x7f46f01460a0 2024-01-21T03:20:02.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.021+0000 7f46ed7fa640 10 client.4244.objecter ms_dispatch 0x7f46f0000b90 osd_map(25..25 src has 1..25) v4 2024-01-21T03:20:02.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.021+0000 7f46ed7fa640 3 client.4244.objecter handle_osd_map got epochs [25,25] > 0 2024-01-21T03:20:02.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.021+0000 7f46ed7fa640 3 client.4244.objecter handle_osd_map decoding full epoch 25 2024-01-21T03:20:02.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.021+0000 7f46ed7fa640 20 client.4244.objecter dump_active .. 0 homeless 2024-01-21T03:20:02.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.021+0000 7f46ed7fa640 1 -- 172.21.15.7:0/1311754282 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f46e8055760 con 0x7f46f01460a0 2024-01-21T03:20:02.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.021+0000 7f46ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:02.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.021+0000 7f46ed7fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:02.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.269+0000 7f46f6ab4640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-01-21T03:20:02.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.269+0000 7f46f6ab4640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-01-21T03:20:02.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.269+0000 7f46f6ab4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:02.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.269+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1311754282 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f46f0142b80 con 0x7f46f01460a0 2024-01-21T03:20:02.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.434+0000 7f46ed7fa640 1 -- 172.21.15.7:0/1311754282 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170761 (secure 0 0 0) 0x7f46f0142b80 con 0x7f46f01460a0 2024-01-21T03:20:02.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.434+0000 7f46ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-01-21T03:20:02.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.434+0000 7f46ed7fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.436+0000 7f46f6ab4640 10 librados: watch_flush enter 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.436+0000 7f46f6ab4640 10 librados: watch_flush exit 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.436+0000 7f46f6ab4640 20 client.4244.objecter shutdown clearing up homeless session... 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.436+0000 7f46f6ab4640 10 client.4244.objecter successfully canceled tick 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.436+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1311754282 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f46d406c860 msgr2=0x7f46d406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.436+0000 7f46f6ab4640 1 --2- 172.21.15.7:0/1311754282 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f46d406c860 0x7f46d406ed20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f46e4036000 tx=0x7f46e4008030 comp rx=0 tx=0).stop 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.436+0000 7f46f6ab4640 10 monclient: shutdown 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.436+0000 7f46f6ab4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.436+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1311754282 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46f01460a0 msgr2=0x7f46f01d6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.436+0000 7f46f6ab4640 1 --2- 172.21.15.7:0/1311754282 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46f01460a0 0x7f46f01d6ee0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f46e800efc0 tx=0x7f46e800c490 comp rx=0 tx=0).stop 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.437+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1311754282 shutdown_connections 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.437+0000 7f46f6ab4640 1 --2- 172.21.15.7:0/1311754282 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f46d406c860 0x7f46d406ed20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.437+0000 7f46f6ab4640 1 --2- 172.21.15.7:0/1311754282 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f46f01460a0 0x7f46f01d6ee0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.437+0000 7f46f6ab4640 1 --2- 172.21.15.7:0/1311754282 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f013f8b0 0x7f46f0142030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.437+0000 7f46f6ab4640 1 --2- 172.21.15.7:0/1311754282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f00a8a40 0x7f46f0144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.437+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1311754282 >> 172.21.15.7:0/1311754282 conn(0x7f46f0066f50 msgr2=0x7f46f013a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.437+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1311754282 shutdown_connections 2024-01-21T03:20:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.437+0000 7f46f6ab4640 1 -- 172.21.15.7:0/1311754282 wait complete. 2024-01-21T03:20:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.437+0000 7f46f6ab4640 1 librados: shutdown 2024-01-21T03:20:02.432 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"active_gid":4103,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2590781548},{"type":"v1","addr":"172.21.15.7:6801","nonce":2590781548}]},"active_addr":"172.21.15.7:6801/2590781548","active_change":"2024-01-21T03:19:03.210998+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[{"gid":4101,"name":"x","mgr_features":4540701272860131327,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2618040297}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3925758181}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3017415968}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":693106086}]}]} 2024-01-21T03:20:02.438 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-21T03:20:02.438 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-21T03:20:02.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-21T03:20:02.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.554+0000 7f259bc21640 10 monclient: get_monmap_and_config 2024-01-21T03:20:02.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.554+0000 7f259bc21640 10 monclient: build_initial_monmap 2024-01-21T03:20:02.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.554+0000 7f259bc21640 10 monclient: monmap: 2024-01-21T03:20:02.540 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:02.540 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:02.540 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:02.555808+0000 2024-01-21T03:20:02.540 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:02.555808+0000 2024-01-21T03:20:02.541 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:02.541 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:02.541 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:02.541 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:02.541 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:02.541 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:02.541 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:02.541 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:02.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.555+0000 7f259bc21640 1 Processor -- start 2024-01-21T03:20:02.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.555+0000 7f259bc21640 1 -- start start 2024-01-21T03:20:02.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.555+0000 7f259bc21640 10 monclient: init 2024-01-21T03:20:02.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:02.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-01-21T03:20:02.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f25940a8a40 0x7f2594145b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:02.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 10 monclient(hunting): picked mon.noname-c con 0x7f25940a8a40 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:02.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25941460a0 0x7f259413f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:02.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 10 monclient(hunting): picked mon.noname-b con 0x7f25941460a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:02.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25940b21c0 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:02.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25940aa9b0 con 0x7f25940b21c0 2024-01-21T03:20:02.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:02.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25940aa7d0 con 0x7f25940a8a40 2024-01-21T03:20:02.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:02.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25940aac90 con 0x7f25941460a0 2024-01-21T03:20:02.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:02.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.556+0000 7f259bc21640 10 monclient(hunting): authenticate will time out at 1348.871984s 2024-01-21T03:20:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.577+0000 7f25999bd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f25940a8a40 0x7f2594145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.577+0000 7f25991bc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25941460a0 0x7f259413f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.578+0000 7f25991bc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25941460a0 0x7f259413f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45746/0 (socket says 172.21.15.7:45746) 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.578+0000 7f25991bc640 1 -- 172.21.15.7:0/131929037 learned_addr learned my addr 172.21.15.7:0/131929037 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.578+0000 7f25991bc640 10 monclient(hunting): get_auth_request con 0x7f25941460a0 auth_method 0 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.578+0000 7f25991bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.578+0000 7f25991bc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.578+0000 7f25991bc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.578+0000 7f25999bd640 10 monclient(hunting): get_auth_request con 0x7f25940a8a40 auth_method 0 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.578+0000 7f25999bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.578+0000 7f25999bd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.578+0000 7f25999bd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.579+0000 7f25991bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.579+0000 7f25991bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.579+0000 7f25991bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.579+0000 7f25999bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.579+0000 7f25999bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.579+0000 7f25999bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.580+0000 7f25991bc640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-01-21T03:20:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.580+0000 7f25991bc640 1 -- 172.21.15.7:0/131929037 >> v2:172.21.15.177:3300/0 conn(0x7f25940a8a40 msgr2=0x7f2594145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.580+0000 7f25991bc640 1 --2- 172.21.15.7:0/131929037 >> v2:172.21.15.177:3300/0 conn(0x7f25940a8a40 0x7f2594145b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.580+0000 7f25991bc640 1 -- 172.21.15.7:0/131929037 >> v1:172.21.15.177:6789/0 conn(0x7f25940b21c0 legacy=0x7f2594141f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.580+0000 7f25991bc640 10 monclient: _finish_hunting 0 2024-01-21T03:20:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.580+0000 7f25991bc640 1 monclient: found mon.noname-b 2024-01-21T03:20:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.580+0000 7f25991bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.580+0000 7f25991bc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.580+0000 7f25991bc640 1 -- 172.21.15.7:0/131929037 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25940aaf70 con 0x7f25941460a0 2024-01-21T03:20:02.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.582+0000 7f25991bc640 1 --2- 172.21.15.7:0/131929037 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25941460a0 0x7f259413f1a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2588009870 tx=0x7f258802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4af1d5c0ea4ae043 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f25989bb640 1 -- 172.21.15.7:0/131929037 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258802f840 con 0x7f25941460a0 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f25989bb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f25989bb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f25989bb640 10 monclient: dump: 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f25989bb640 10 monclient: _finish_auth 0 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f25989bb640 10 monclient: _check_auth_tickets 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f25989bb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f25989bb640 1 -- 172.21.15.7:0/131929037 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258802f9a0 con 0x7f25941460a0 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f259bc21640 5 monclient: authenticate success, global_id 4247 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f259bc21640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f25989bb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f259bc21640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f25989bb640 1 -- 172.21.15.7:0/131929037 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258802fd30 con 0x7f25941460a0 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f25989bb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f259bc21640 4 set_mon_vals no callback set 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f25989bb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f25989bb640 10 monclient: dump: 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f259bc21640 10 monclient: shutdown 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f259bc21640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f259bc21640 1 -- 172.21.15.7:0/131929037 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25941460a0 msgr2=0x7f259413f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f259bc21640 1 --2- 172.21.15.7:0/131929037 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25941460a0 0x7f259413f1a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2588009870 tx=0x7f258802ba20 comp rx=0 tx=0).stop 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f259bc21640 1 -- 172.21.15.7:0/131929037 shutdown_connections 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.586+0000 7f259bc21640 1 --2- 172.21.15.7:0/131929037 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25941460a0 0x7f259413f1a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.587+0000 7f259bc21640 1 --2- 172.21.15.7:0/131929037 >> v2:172.21.15.177:3300/0 conn(0x7f25940a8a40 0x7f2594145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.587+0000 7f259bc21640 1 -- 172.21.15.7:0/131929037 >> 172.21.15.7:0/131929037 conn(0x7f2594066f50 msgr2=0x7f25940b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:02.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.587+0000 7f259bc21640 1 -- 172.21.15.7:0/131929037 shutdown_connections 2024-01-21T03:20:02.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.587+0000 7f259bc21640 1 -- 172.21.15.7:0/131929037 wait complete. 2024-01-21T03:20:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.587+0000 7f259bc21640 10 monclient: build_initial_monmap 2024-01-21T03:20:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.587+0000 7f259bc21640 10 monclient: monmap: 2024-01-21T03:20:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.587+0000 7f259bc21640 1 librados: starting msgr at 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 1 librados: starting objecter 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 1 Processor -- start 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 1 -- start start 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 1 librados: setting wanted keys 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 1 librados: calling monclient init 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 10 monclient: init 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25940a8a40 0x7f25941d76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 10 monclient(hunting): picked mon.noname-c con 0x7f25940a8a40 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25940b21c0 0x7f25940b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 10 monclient(hunting): picked mon.noname-a con 0x7f25940b21c0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25941460a0 0x7f25941424c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 10 monclient(hunting): picked mon.noname-b con 0x7f25941460a0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25940ac4d0 con 0x7f25940b21c0 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25941d7d40 con 0x7f25941460a0 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25941d81d0 con 0x7f25940a8a40 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.588+0000 7f259bc21640 10 monclient(hunting): authenticate will time out at 1348.904463s 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25999bd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25940a8a40 0x7f25941d76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25991bc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25940b21c0 0x7f25940b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25999bd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25940a8a40 0x7f25941d76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45762/0 (socket says 172.21.15.7:45762) 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25999bd640 1 -- 172.21.15.7:0/2682777199 learned_addr learned my addr 172.21.15.7:0/2682777199 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25999bd640 10 monclient(hunting): get_auth_request con 0x7f25940a8a40 auth_method 0 2024-01-21T03:20:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25999bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25999bd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25999bd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25991bc640 10 monclient(hunting): get_auth_request con 0x7f25940b21c0 auth_method 0 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25991bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25991bc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25991bc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25999bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25999bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.589+0000 7f25999bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.590+0000 7f25991bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.590+0000 7f25991bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.590+0000 7f25991bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.591+0000 7f259a1be640 1 --2- 172.21.15.7:0/2682777199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25941460a0 0x7f25941424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f259a1be640 10 monclient(hunting): get_auth_request con 0x7f25941460a0 auth_method 0 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f259a1be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f259a1be640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f259a1be640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f259a1be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f259a1be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f259a1be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f25999bd640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-01-21T03:20:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f25999bd640 1 -- 172.21.15.7:0/2682777199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25941460a0 msgr2=0x7f25941424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f25999bd640 1 --2- 172.21.15.7:0/2682777199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25941460a0 0x7f25941424c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f25999bd640 1 -- 172.21.15.7:0/2682777199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25940b21c0 msgr2=0x7f25940b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f25999bd640 1 --2- 172.21.15.7:0/2682777199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25940b21c0 0x7f25940b09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f25999bd640 10 monclient: _finish_hunting 0 2024-01-21T03:20:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f25999bd640 1 monclient: found mon.noname-c 2024-01-21T03:20:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f25999bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f25999bd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:20:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.592+0000 7f25999bd640 1 -- 172.21.15.7:0/2682777199 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25880094d0 con 0x7f25940a8a40 2024-01-21T03:20:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.595+0000 7f25999bd640 1 --2- 172.21.15.7:0/2682777199 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25940a8a40 0x7f25941d76d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f258c009db0 tx=0x7f258c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.595+0000 7f25927fc640 10 client.?.objecter ms_handle_connect 0x7f25940a8a40 2024-01-21T03:20:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.595+0000 7f25927fc640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f25927fc640 1 -- 172.21.15.7:0/2682777199 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c014480 con 0x7f25940a8a40 2024-01-21T03:20:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f25927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f25927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f25927fc640 10 monclient: dump: 2024-01-21T03:20:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f25927fc640 10 monclient: _finish_auth 0 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f25927fc640 10 monclient: _check_auth_tickets 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f25927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f25927fc640 1 -- 172.21.15.7:0/2682777199 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c00f040 con 0x7f25940a8a40 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f259bc21640 5 monclient: authenticate success, global_id 4250 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f259bc21640 10 monclient: _renew_subs 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f259bc21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f259bc21640 1 -- 172.21.15.7:0/2682777199 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25941d83f0 con 0x7f25940a8a40 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f25927fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f25927fc640 1 -- 172.21.15.7:0/2682777199 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c014800 con 0x7f25940a8a40 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f25927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f25927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f25927fc640 10 monclient: dump: 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f2593fff640 4 set_mon_vals no callback set 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f259bc21640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f259bc21640 10 monclient: _renew_subs 2024-01-21T03:20:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f259bc21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:02.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f259bc21640 1 -- 172.21.15.7:0/2682777199 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25941dadb0 con 0x7f25940a8a40 2024-01-21T03:20:02.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.599+0000 7f259bc21640 1 librados: init done 2024-01-21T03:20:02.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.600+0000 7f259bc21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:02.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.600+0000 7f259bc21640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:02.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.600+0000 7f259bc21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:02.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.600+0000 7f259bc21640 1 -- 172.21.15.7:0/2682777199 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259404e970 con 0x7f25940a8a40 2024-01-21T03:20:02.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.861+0000 7f25927fc640 1 -- 172.21.15.7:0/2682777199 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f258c014960 con 0x7f25940a8a40 2024-01-21T03:20:02.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.861+0000 7f25927fc640 1 --2- 172.21.15.7:0/2682777199 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f258006c8b0 0x7f258006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:02.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.861+0000 7f25991bc640 1 --2- 172.21.15.7:0/2682777199 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f258006c8b0 0x7f258006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:02.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.861+0000 7f25991bc640 10 monclient: get_auth_request con 0x7f258006c8b0 auth_method 0 2024-01-21T03:20:02.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.862+0000 7f25991bc640 1 --2- 172.21.15.7:0/2682777199 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f258006c8b0 0x7f258006ed70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2588036000 tx=0x7f2588008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:02.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.862+0000 7f25927fc640 10 client.4250.objecter ms_handle_connect 0x7f258006c8b0 2024-01-21T03:20:02.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.875+0000 7f25927fc640 1 -- 172.21.15.7:0/2682777199 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f258c059500 con 0x7f25940a8a40 2024-01-21T03:20:02.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.875+0000 7f25927fc640 10 client.4250.objecter ms_dispatch 0x7f2594000b90 osd_map(25..25 src has 1..25) v4 2024-01-21T03:20:02.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.875+0000 7f25927fc640 3 client.4250.objecter handle_osd_map got epochs [25,25] > 0 2024-01-21T03:20:02.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.875+0000 7f25927fc640 3 client.4250.objecter handle_osd_map decoding full epoch 25 2024-01-21T03:20:02.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.875+0000 7f25927fc640 20 client.4250.objecter dump_active .. 0 homeless 2024-01-21T03:20:02.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.875+0000 7f25927fc640 1 -- 172.21.15.7:0/2682777199 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f258c055840 con 0x7f25940a8a40 2024-01-21T03:20:02.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.875+0000 7f25927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:02.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:02.875+0000 7f25927fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:03.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.100+0000 7f259bc21640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:20:03.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.100+0000 7f259bc21640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:20:03.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.100+0000 7f259bc21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:03.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.100+0000 7f259bc21640 1 -- 172.21.15.7:0/2682777199 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25940b42a0 con 0x7f25940a8a40 2024-01-21T03:20:03.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.202+0000 7f25927fc640 1 -- 172.21.15.7:0/2682777199 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+14615 (secure 0 0 0) 0x7f258c058e60 con 0x7f25940a8a40 2024-01-21T03:20:03.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.202+0000 7f25927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:20:03.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.202+0000 7f25927fc640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:03.189 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:20:03.189 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":26,"fsid":"8de9e7dd-e982-4048-83dd-ee52350dd56d","created":"2024-01-21T03:18:57.607470+0000","modified":"2024-01-21T03:20:02.620274+0000","last_up_change":"2024-01-21T03:19:56.402389+0000","last_in_change":"2024-01-21T03:19:02.952334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-21T03:19:44.021407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-21T03:19:59.115939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f4767aa-192e-4ec9-addf-a51c0872da7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6819","nonce":2019292672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6821","nonce":2019292672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6825","nonce":2019292672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6823","nonce":2019292672}]},"public_addr":"172.21.15.7:6819/2019292672","cluster_addr":"172.21.15.7:6821/2019292672","heartbeat_back_addr":"172.21.15.7:6825/2019292672","heartbeat_front_addr":"172.21.15.7:6823/2019292672","state":["exists","up"]},{"osd":1,"uuid":"066d7378-561f-4e74-b97b-2ead2fa2625e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6811","nonce":2176696455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6813","nonce":2176696455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6817","nonce":2176696455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6815","nonce":2176696455}]},"public_addr":"172.21.15.7:6811/2176696455","cluster_addr":"172.21.15.7:6813/2176696455","heartbeat_back_addr":"172.21.15.7:6817/2176696455","heartbeat_front_addr":"172.21.15.7:6815/2176696455","state":["exists","up"]},{"osd":2,"uuid":"f219649b-4d50-40a8-b74e-5a7f16b3c8c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6827","nonce":888884750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6829","nonce":888884750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6833","nonce":888884750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6831","nonce":888884750}]},"public_addr":"172.21.15.7:6827/888884750","cluster_addr":"172.21.15.7:6829/888884750","heartbeat_back_addr":"172.21.15.7:6833/888884750","heartbeat_front_addr":"172.21.15.7:6831/888884750","state":["exists","up"]},{"osd":3,"uuid":"6379a402-760a-4f07-9491-0918752a18fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6803","nonce":1182044719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6805","nonce":1182044719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6809","nonce":1182044719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6807","nonce":1182044719}]},"public_addr":"172.21.15.7:6803/1182044719","cluster_addr":"172.21.15.7:6805/1182044719","heartbeat_back_addr":"172.21.15.7:6809/1182044719","heartbeat_front_addr":"172.21.15.7:6807/1182044719","state":["exists","up"]},{"osd":4,"uuid":"dc3602e1-b59e-4489-9882-40c914e2a0d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6809","nonce":3975537014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6811","nonce":3975537014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6815","nonce":3975537014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6813","nonce":3975537014}]},"public_addr":"172.21.15.177:6809/3975537014","cluster_addr":"172.21.15.177:6811/3975537014","heartbeat_back_addr":"172.21.15.177:6815/3975537014","heartbeat_front_addr":"172.21.15.177:6813/3975537014","state":["exists","up"]},{"osd":5,"uuid":"d42afb34-e1ca-4b21-81e9-78d1c8ea454e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6817","nonce":2318383718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6819","nonce":2318383718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6824","nonce":2318383718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6821","nonce":2318383718}]},"public_addr":"172.21.15.177:6817/2318383718","cluster_addr":"172.21.15.177:6819/2318383718","heartbeat_back_addr":"172.21.15.177:6824/2318383718","heartbeat_front_addr":"172.21.15.177:6821/2318383718","state":["exists","up"]},{"osd":6,"uuid":"1aef2850-8b41-4d39-859e-ed9c4df434b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6801","nonce":1092304531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6803","nonce":1092304531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6807","nonce":1092304531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6805","nonce":1092304531}]},"public_addr":"172.21.15.177:6801/1092304531","cluster_addr":"172.21.15.177:6803/1092304531","heartbeat_back_addr":"172.21.15.177:6807/1092304531","heartbeat_front_addr":"172.21.15.177:6805/1092304531","state":["exists","up"]},{"osd":7,"uuid":"8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6823","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6825","nonce":4069998673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6827","nonce":4069998673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6831","nonce":4069998673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6829","nonce":4069998673}]},"public_addr":"172.21.15.177:6825/4069998673","cluster_addr":"172.21.15.177:6827/4069998673","heartbeat_back_addr":"172.21.15.177:6831/4069998673","heartbeat_front_addr":"172.21.15.177:6829/4069998673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:40.924122+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:41.225506+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:41.727729+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:41.835880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:38.955717+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:38.638101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:39.011972+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:39.042445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T03:20:03.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 10 librados: watch_flush enter 2024-01-21T03:20:03.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 10 librados: watch_flush exit 2024-01-21T03:20:03.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 20 client.4250.objecter shutdown clearing up homeless session... 2024-01-21T03:20:03.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 10 client.4250.objecter successfully canceled tick 2024-01-21T03:20:03.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 1 -- 172.21.15.7:0/2682777199 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f258006c8b0 msgr2=0x7f258006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:03.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 1 --2- 172.21.15.7:0/2682777199 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f258006c8b0 0x7f258006ed70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2588036000 tx=0x7f2588008030 comp rx=0 tx=0).stop 2024-01-21T03:20:03.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 10 monclient: shutdown 2024-01-21T03:20:03.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:03.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 1 -- 172.21.15.7:0/2682777199 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25940a8a40 msgr2=0x7f25941d76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:03.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 1 --2- 172.21.15.7:0/2682777199 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25940a8a40 0x7f25941d76d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f258c009db0 tx=0x7f258c00c490 comp rx=0 tx=0).stop 2024-01-21T03:20:03.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 1 -- 172.21.15.7:0/2682777199 shutdown_connections 2024-01-21T03:20:03.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 1 --2- 172.21.15.7:0/2682777199 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f258006c8b0 0x7f258006ed70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 1 --2- 172.21.15.7:0/2682777199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25941460a0 0x7f25941424c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 1 --2- 172.21.15.7:0/2682777199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25940b21c0 0x7f25940b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 1 --2- 172.21.15.7:0/2682777199 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f25940a8a40 0x7f25941d76d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 1 -- 172.21.15.7:0/2682777199 >> 172.21.15.7:0/2682777199 conn(0x7f2594066f50 msgr2=0x7f259413de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:03.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 1 -- 172.21.15.7:0/2682777199 shutdown_connections 2024-01-21T03:20:03.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.204+0000 7f259bc21640 1 -- 172.21.15.7:0/2682777199 wait complete. 2024-01-21T03:20:03.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.205+0000 7f259bc21640 1 librados: shutdown 2024-01-21T03:20:03.202 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-21T03:20:03.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-21T03:20:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.316+0000 7f50840ec640 10 monclient: get_monmap_and_config 2024-01-21T03:20:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.316+0000 7f50840ec640 10 monclient: build_initial_monmap 2024-01-21T03:20:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.316+0000 7f50840ec640 10 monclient: monmap: 2024-01-21T03:20:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:03.317757+0000 2024-01-21T03:20:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:03.317757+0000 2024-01-21T03:20:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:03.304 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:03.304 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:03.304 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:03.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.317+0000 7f50840ec640 1 Processor -- start 2024-01-21T03:20:03.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.317+0000 7f50840ec640 1 -- start start 2024-01-21T03:20:03.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.317+0000 7f50840ec640 10 monclient: init 2024-01-21T03:20:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-01-21T03:20:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f507c0a8a40 0x7f507c145a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 10 monclient(hunting): picked mon.noname-c con 0x7f507c0a8a40 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f507c0a8e20 0x7f507c145fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 10 monclient(hunting): picked mon.noname-b con 0x7f507c0a8e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f507c13f110 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507c0aa9b0 con 0x7f507c13f110 2024-01-21T03:20:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507c0aac90 con 0x7f507c0a8a40 2024-01-21T03:20:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0aaf70 con 0x7f507c0a8e20 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f50840ec640 10 monclient(hunting): authenticate will time out at 1349.633955s 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.318+0000 7f5081687640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f507c0a8e20 0x7f507c145fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081687640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f507c0a8e20 0x7f507c145fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45776/0 (socket says 172.21.15.7:45776) 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081687640 1 -- 172.21.15.7:0/1219327920 learned_addr learned my addr 172.21.15.7:0/1219327920 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081687640 10 monclient(hunting): get_auth_request con 0x7f507c0a8e20 auth_method 0 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081687640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081687640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081687640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081e88640 1 --2- 172.21.15.7:0/1219327920 >> v2:172.21.15.177:3300/0 conn(0x7f507c0a8a40 0x7f507c145a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081e88640 10 monclient(hunting): get_auth_request con 0x7f507c0a8a40 auth_method 0 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081e88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081e88640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081e88640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081687640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081687640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.319+0000 7f5081687640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.320+0000 7f5081e88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.320+0000 7f5081e88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.320+0000 7f5081e88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.321+0000 7f5081687640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-01-21T03:20:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.321+0000 7f5081687640 1 -- 172.21.15.7:0/1219327920 >> v2:172.21.15.177:3300/0 conn(0x7f507c0a8a40 msgr2=0x7f507c145a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.321+0000 7f5081687640 1 --2- 172.21.15.7:0/1219327920 >> v2:172.21.15.177:3300/0 conn(0x7f507c0a8a40 0x7f507c145a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.321+0000 7f5081687640 1 -- 172.21.15.7:0/1219327920 >> v1:172.21.15.177:6789/0 conn(0x7f507c13f110 legacy=0x7f507c13f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.321+0000 7f5081687640 10 monclient: _finish_hunting 0 2024-01-21T03:20:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.321+0000 7f5081687640 1 monclient: found mon.noname-b 2024-01-21T03:20:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.321+0000 7f5081687640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.321+0000 7f5081687640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.321+0000 7f5081687640 1 -- 172.21.15.7:0/1219327920 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c145560 con 0x7f507c0a8e20 2024-01-21T03:20:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.323+0000 7f5081687640 1 --2- 172.21.15.7:0/1219327920 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f507c0a8e20 0x7f507c145fc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5070009870 tx=0x7f507002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=888891d55a1c3c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f5080e86640 1 -- 172.21.15.7:0/1219327920 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070031640 con 0x7f507c0a8e20 2024-01-21T03:20:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f5080e86640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f5080e86640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f5080e86640 10 monclient: dump: 2024-01-21T03:20:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f5080e86640 10 monclient: _finish_auth 0 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f5080e86640 10 monclient: _check_auth_tickets 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f5080e86640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f5080e86640 1 -- 172.21.15.7:0/1219327920 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50700317a0 con 0x7f507c0a8e20 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f5080e86640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f5080e86640 1 -- 172.21.15.7:0/1219327920 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070031b30 con 0x7f507c0a8e20 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f5080e86640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f5080e86640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f5080e86640 10 monclient: dump: 2024-01-21T03:20:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:03.315 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f50840ec640 5 monclient: authenticate success, global_id 4253 2024-01-21T03:20:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f50840ec640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.327+0000 7f50840ec640 4 set_mon_vals no callback set 2024-01-21T03:20:03.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.328+0000 7f50840ec640 10 monclient: shutdown 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.328+0000 7f50840ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.328+0000 7f50840ec640 1 -- 172.21.15.7:0/1219327920 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f507c0a8e20 msgr2=0x7f507c145fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.328+0000 7f50840ec640 1 --2- 172.21.15.7:0/1219327920 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f507c0a8e20 0x7f507c145fc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5070009870 tx=0x7f507002b8f0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.328+0000 7f50840ec640 1 -- 172.21.15.7:0/1219327920 shutdown_connections 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.328+0000 7f50840ec640 1 --2- 172.21.15.7:0/1219327920 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f507c0a8e20 0x7f507c145fc0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.328+0000 7f50840ec640 1 --2- 172.21.15.7:0/1219327920 >> v2:172.21.15.177:3300/0 conn(0x7f507c0a8a40 0x7f507c145a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.328+0000 7f50840ec640 1 -- 172.21.15.7:0/1219327920 >> 172.21.15.7:0/1219327920 conn(0x7f507c066eb0 msgr2=0x7f507c0b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.328+0000 7f50840ec640 1 -- 172.21.15.7:0/1219327920 shutdown_connections 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.328+0000 7f50840ec640 1 -- 172.21.15.7:0/1219327920 wait complete. 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.328+0000 7f50840ec640 10 monclient: build_initial_monmap 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.328+0000 7f50840ec640 10 monclient: monmap: 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:03.320 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:03.320 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:03.320 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:03.320 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:03.320 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:03.320 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:03.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 1 librados: starting msgr at 2024-01-21T03:20:03.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 1 librados: starting objecter 2024-01-21T03:20:03.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 1 Processor -- start 2024-01-21T03:20:03.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 1 -- start start 2024-01-21T03:20:03.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 1 librados: setting wanted keys 2024-01-21T03:20:03.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 1 librados: calling monclient init 2024-01-21T03:20:03.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 10 monclient: init 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f507c0a8a40 0x7f507c134640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 10 monclient(hunting): picked mon.noname-a con 0x7f507c0a8a40 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f507c0a8e20 0x7f507c1daad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 10 monclient(hunting): picked mon.noname-b con 0x7f507c0a8e20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f507c13f110 0x7f507c1db010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.329+0000 7f50840ec640 10 monclient(hunting): picked mon.noname-c con 0x7f507c13f110 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f50840ec640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f50840ec640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c1383b0 con 0x7f507c0a8a40 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f50840ec640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f50840ec640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0b72b0 con 0x7f507c0a8e20 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f50840ec640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f50840ec640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c1db550 con 0x7f507c13f110 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f50840ec640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:03.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f50840ec640 10 monclient(hunting): authenticate will time out at 1349.645765s 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f5082689640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f507c13f110 0x7f507c1db010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f5081e88640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f507c0a8a40 0x7f507c134640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f5081687640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f507c0a8e20 0x7f507c1daad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f5082689640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f507c13f110 0x7f507c1db010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45786/0 (socket says 172.21.15.7:45786) 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f5082689640 1 -- 172.21.15.7:0/2833320706 learned_addr learned my addr 172.21.15.7:0/2833320706 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f5082689640 10 monclient(hunting): get_auth_request con 0x7f507c13f110 auth_method 0 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f5082689640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f5082689640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f5082689640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f5081687640 10 monclient(hunting): get_auth_request con 0x7f507c0a8e20 auth_method 0 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f5081687640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f5081687640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.330+0000 7f5081687640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.331+0000 7f5081e88640 10 monclient(hunting): get_auth_request con 0x7f507c0a8a40 auth_method 0 2024-01-21T03:20:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.331+0000 7f5081e88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.331+0000 7f5081e88640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.331+0000 7f5081e88640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.331+0000 7f5081687640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.331+0000 7f5081687640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.331+0000 7f5081687640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.331+0000 7f5082689640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.331+0000 7f5082689640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.331+0000 7f5082689640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.331+0000 7f5081e88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.331+0000 7f5081e88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.331+0000 7f5081e88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.333+0000 7f5081687640 10 monclient(hunting): handle_auth_done global_id 4285 payload 824 2024-01-21T03:20:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.334+0000 7f5081687640 1 -- 172.21.15.7:0/2833320706 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f507c13f110 msgr2=0x7f507c1db010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.334+0000 7f5081687640 1 --2- 172.21.15.7:0/2833320706 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f507c13f110 0x7f507c1db010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.334+0000 7f5081687640 1 -- 172.21.15.7:0/2833320706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f507c0a8a40 msgr2=0x7f507c134640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.334+0000 7f5081687640 1 --2- 172.21.15.7:0/2833320706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f507c0a8a40 0x7f507c134640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.334+0000 7f5081687640 10 monclient: _finish_hunting 0 2024-01-21T03:20:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.334+0000 7f5081687640 1 monclient: found mon.noname-b 2024-01-21T03:20:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.334+0000 7f5081687640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.334+0000 7f5081687640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:03.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.334+0000 7f5081687640 1 -- 172.21.15.7:0/2833320706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50700094d0 con 0x7f507c0a8e20 2024-01-21T03:20:03.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.336+0000 7f5081687640 1 --2- 172.21.15.7:0/2833320706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f507c0a8e20 0x7f507c1daad0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5070033000 tx=0x7f50700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:03.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.336+0000 7f507a7fc640 10 client.?.objecter ms_handle_connect 0x7f507c0a8e20 2024-01-21T03:20:03.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.336+0000 7f507a7fc640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:03.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507a7fc640 1 -- 172.21.15.7:0/2833320706 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507003a9f0 con 0x7f507c0a8e20 2024-01-21T03:20:03.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:03.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:03.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507a7fc640 10 monclient: dump: 2024-01-21T03:20:03.326 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:03.326 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507a7fc640 10 monclient: _finish_auth 0 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507a7fc640 10 monclient: _check_auth_tickets 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507a7fc640 1 -- 172.21.15.7:0/2833320706 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507003ab90 con 0x7f507c0a8e20 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f50840ec640 5 monclient: authenticate success, global_id 4285 2024-01-21T03:20:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f50840ec640 10 monclient: _renew_subs 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507a7fc640 1 -- 172.21.15.7:0/2833320706 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50700443f0 con 0x7f507c0a8e20 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f50840ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f50840ec640 1 -- 172.21.15.7:0/2833320706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507c0b24b0 con 0x7f507c0a8e20 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507a7fc640 10 monclient: dump: 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:03.329 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:03.329 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:03.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f507b7fe640 4 set_mon_vals no callback set 2024-01-21T03:20:03.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f50840ec640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:03.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f50840ec640 10 monclient: _renew_subs 2024-01-21T03:20:03.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f50840ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:03.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f50840ec640 1 -- 172.21.15.7:0/2833320706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507c1371c0 con 0x7f507c0a8e20 2024-01-21T03:20:03.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.339+0000 7f50840ec640 1 librados: init done 2024-01-21T03:20:03.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.340+0000 7f50840ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:03.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.340+0000 7f50840ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:03.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.340+0000 7f50840ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:03.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.340+0000 7f50840ec640 1 -- 172.21.15.7:0/2833320706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507c0b70f0 con 0x7f507c0a8e20 2024-01-21T03:20:03.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.357+0000 7f507a7fc640 1 -- 172.21.15.7:0/2833320706 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f507002f050 con 0x7f507c0a8e20 2024-01-21T03:20:03.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.358+0000 7f507a7fc640 1 --2- 172.21.15.7:0/2833320706 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f506806c8b0 0x7f506806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:03.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.358+0000 7f5081e88640 1 --2- 172.21.15.7:0/2833320706 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f506806c8b0 0x7f506806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:03.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.358+0000 7f5081e88640 10 monclient: get_auth_request con 0x7f506806c8b0 auth_method 0 2024-01-21T03:20:03.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.358+0000 7f507a7fc640 10 client.4285.objecter ms_handle_connect 0x7f506806c8b0 2024-01-21T03:20:03.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.358+0000 7f5081e88640 1 --2- 172.21.15.7:0/2833320706 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f506806c8b0 0x7f506806ed70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5074009b40 tx=0x7f5074009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.360+0000 7f507a7fc640 1 -- 172.21.15.7:0/2833320706 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f507007a3e0 con 0x7f507c0a8e20 2024-01-21T03:20:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.360+0000 7f507a7fc640 10 client.4285.objecter ms_dispatch 0x7f507c000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.360+0000 7f507a7fc640 3 client.4285.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.360+0000 7f507a7fc640 3 client.4285.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.360+0000 7f507a7fc640 20 client.4285.objecter dump_active .. 0 homeless 2024-01-21T03:20:03.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.594+0000 7f507a7fc640 1 -- 172.21.15.7:0/2833320706 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5070076720 con 0x7f507c0a8e20 2024-01-21T03:20:03.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.595+0000 7f507a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:03.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.595+0000 7f507a7fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:03.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.789+0000 7f50840ec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:20:03.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.789+0000 7f50840ec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:20:03.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.789+0000 7f50840ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:03.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.789+0000 7f50840ec640 1 -- 172.21.15.7:0/2833320706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f507c04e970 con 0x7f507c0a8e20 2024-01-21T03:20:03.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.823+0000 7f507a7fc640 1 -- 172.21.15.7:0/2833320706 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+14615 (secure 0 0 0) 0x7f5070079d40 con 0x7f507c0a8e20 2024-01-21T03:20:03.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.823+0000 7f507a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:20:03.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.823+0000 7f507a7fc640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:03.809 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:20:03.809 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":26,"fsid":"8de9e7dd-e982-4048-83dd-ee52350dd56d","created":"2024-01-21T03:18:57.607470+0000","modified":"2024-01-21T03:20:02.620274+0000","last_up_change":"2024-01-21T03:19:56.402389+0000","last_in_change":"2024-01-21T03:19:02.952334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-21T03:19:44.021407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-21T03:19:59.115939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f4767aa-192e-4ec9-addf-a51c0872da7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6819","nonce":2019292672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6821","nonce":2019292672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6825","nonce":2019292672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6823","nonce":2019292672}]},"public_addr":"172.21.15.7:6819/2019292672","cluster_addr":"172.21.15.7:6821/2019292672","heartbeat_back_addr":"172.21.15.7:6825/2019292672","heartbeat_front_addr":"172.21.15.7:6823/2019292672","state":["exists","up"]},{"osd":1,"uuid":"066d7378-561f-4e74-b97b-2ead2fa2625e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6811","nonce":2176696455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6813","nonce":2176696455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6817","nonce":2176696455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6815","nonce":2176696455}]},"public_addr":"172.21.15.7:6811/2176696455","cluster_addr":"172.21.15.7:6813/2176696455","heartbeat_back_addr":"172.21.15.7:6817/2176696455","heartbeat_front_addr":"172.21.15.7:6815/2176696455","state":["exists","up"]},{"osd":2,"uuid":"f219649b-4d50-40a8-b74e-5a7f16b3c8c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6827","nonce":888884750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6829","nonce":888884750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6833","nonce":888884750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6831","nonce":888884750}]},"public_addr":"172.21.15.7:6827/888884750","cluster_addr":"172.21.15.7:6829/888884750","heartbeat_back_addr":"172.21.15.7:6833/888884750","heartbeat_front_addr":"172.21.15.7:6831/888884750","state":["exists","up"]},{"osd":3,"uuid":"6379a402-760a-4f07-9491-0918752a18fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6803","nonce":1182044719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6805","nonce":1182044719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6809","nonce":1182044719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6807","nonce":1182044719}]},"public_addr":"172.21.15.7:6803/1182044719","cluster_addr":"172.21.15.7:6805/1182044719","heartbeat_back_addr":"172.21.15.7:6809/1182044719","heartbeat_front_addr":"172.21.15.7:6807/1182044719","state":["exists","up"]},{"osd":4,"uuid":"dc3602e1-b59e-4489-9882-40c914e2a0d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6809","nonce":3975537014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6811","nonce":3975537014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6815","nonce":3975537014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6813","nonce":3975537014}]},"public_addr":"172.21.15.177:6809/3975537014","cluster_addr":"172.21.15.177:6811/3975537014","heartbeat_back_addr":"172.21.15.177:6815/3975537014","heartbeat_front_addr":"172.21.15.177:6813/3975537014","state":["exists","up"]},{"osd":5,"uuid":"d42afb34-e1ca-4b21-81e9-78d1c8ea454e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6817","nonce":2318383718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6819","nonce":2318383718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6824","nonce":2318383718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6821","nonce":2318383718}]},"public_addr":"172.21.15.177:6817/2318383718","cluster_addr":"172.21.15.177:6819/2318383718","heartbeat_back_addr":"172.21.15.177:6824/2318383718","heartbeat_front_addr":"172.21.15.177:6821/2318383718","state":["exists","up"]},{"osd":6,"uuid":"1aef2850-8b41-4d39-859e-ed9c4df434b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6801","nonce":1092304531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6803","nonce":1092304531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6807","nonce":1092304531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6805","nonce":1092304531}]},"public_addr":"172.21.15.177:6801/1092304531","cluster_addr":"172.21.15.177:6803/1092304531","heartbeat_back_addr":"172.21.15.177:6807/1092304531","heartbeat_front_addr":"172.21.15.177:6805/1092304531","state":["exists","up"]},{"osd":7,"uuid":"8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6823","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6825","nonce":4069998673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6827","nonce":4069998673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6831","nonce":4069998673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6829","nonce":4069998673}]},"public_addr":"172.21.15.177:6825/4069998673","cluster_addr":"172.21.15.177:6827/4069998673","heartbeat_back_addr":"172.21.15.177:6831/4069998673","heartbeat_front_addr":"172.21.15.177:6829/4069998673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:40.924122+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:41.225506+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:41.727729+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:41.835880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:38.955717+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:38.638101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:39.011972+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:39.042445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T03:20:03.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 10 librados: watch_flush enter 2024-01-21T03:20:03.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 10 librados: watch_flush exit 2024-01-21T03:20:03.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 20 client.4285.objecter shutdown clearing up homeless session... 2024-01-21T03:20:03.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 10 client.4285.objecter successfully canceled tick 2024-01-21T03:20:03.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 1 -- 172.21.15.7:0/2833320706 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f506806c8b0 msgr2=0x7f506806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:03.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 1 --2- 172.21.15.7:0/2833320706 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f506806c8b0 0x7f506806ed70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5074009b40 tx=0x7f5074009190 comp rx=0 tx=0).stop 2024-01-21T03:20:03.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 10 monclient: shutdown 2024-01-21T03:20:03.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:03.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 1 -- 172.21.15.7:0/2833320706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f507c0a8e20 msgr2=0x7f507c1daad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:03.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 1 --2- 172.21.15.7:0/2833320706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f507c0a8e20 0x7f507c1daad0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5070033000 tx=0x7f50700046c0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 1 -- 172.21.15.7:0/2833320706 shutdown_connections 2024-01-21T03:20:03.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 1 --2- 172.21.15.7:0/2833320706 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f506806c8b0 0x7f506806ed70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 1 --2- 172.21.15.7:0/2833320706 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f507c13f110 0x7f507c1db010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 1 --2- 172.21.15.7:0/2833320706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f507c0a8e20 0x7f507c1daad0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 1 --2- 172.21.15.7:0/2833320706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f507c0a8a40 0x7f507c134640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:03.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 1 -- 172.21.15.7:0/2833320706 >> 172.21.15.7:0/2833320706 conn(0x7f507c066eb0 msgr2=0x7f507c13a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:03.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 1 -- 172.21.15.7:0/2833320706 shutdown_connections 2024-01-21T03:20:03.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 1 -- 172.21.15.7:0/2833320706 wait complete. 2024-01-21T03:20:03.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:03.825+0000 7f50840ec640 1 librados: shutdown 2024-01-21T03:20:03.822 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-21T03:20:03.822 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-21T03:20:03.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-21T03:20:03.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-21T03:20:03.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-21T03:20:03.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-21T03:20:03.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-21T03:20:03.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-21T03:20:03.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.008+0000 7f189d62e640 10 monclient: get_monmap_and_config 2024-01-21T03:20:03.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.008+0000 7f3eece00640 10 monclient: get_monmap_and_config 2024-01-21T03:20:03.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.008+0000 7f3eece00640 10 monclient: build_initial_monmap 2024-01-21T03:20:03.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.008+0000 7f3eece00640 10 monclient: monmap: 2024-01-21T03:20:03.995 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:03.995 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:03.995 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.009059+0000 2024-01-21T03:20:03.995 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.009059+0000 2024-01-21T03:20:03.995 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:03.995 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:03.995 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:03.995 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:03.995 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:03.996 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:03.996 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:03.996 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:03.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.008+0000 7f189d62e640 10 monclient: build_initial_monmap 2024-01-21T03:20:03.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.008+0000 7f189d62e640 10 monclient: monmap: 2024-01-21T03:20:03.996 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:03.996 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:03.996 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.009693+0000 2024-01-21T03:20:03.996 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.009693+0000 2024-01-21T03:20:03.997 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:03.997 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:03.997 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:03.997 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:03.997 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:03.997 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:03.997 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:03.997 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:03.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.009+0000 7f6aa1840640 10 monclient: get_monmap_and_config 2024-01-21T03:20:03.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.009+0000 7f6aa1840640 10 monclient: build_initial_monmap 2024-01-21T03:20:03.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.009+0000 7f6aa1840640 10 monclient: monmap: 2024-01-21T03:20:03.998 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:03.998 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:03.998 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.010060+0000 2024-01-21T03:20:03.998 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.010060+0000 2024-01-21T03:20:03.998 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:03.998 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:03.998 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:03.998 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:03.998 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:03.998 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:03.998 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:03.998 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:03.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.009+0000 7f3eece00640 1 Processor -- start 2024-01-21T03:20:03.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.009+0000 7f8275c46640 10 monclient: get_monmap_and_config 2024-01-21T03:20:03.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.009+0000 7f8275c46640 10 monclient: build_initial_monmap 2024-01-21T03:20:03.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.009+0000 7f8275c46640 10 monclient: monmap: 2024-01-21T03:20:03.999 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:03.999 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:03.999 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.010395+0000 2024-01-21T03:20:03.999 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.010395+0000 2024-01-21T03:20:03.999 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:03.999 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.000 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:04.000 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:04.000 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:04.000 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:04.000 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:04.000 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.009+0000 7f3eece00640 1 -- start start 2024-01-21T03:20:04.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.009+0000 7f3eece00640 10 monclient: init 2024-01-21T03:20:04.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f6aa1840640 1 Processor -- start 2024-01-21T03:20:04.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f6aa1840640 1 -- start start 2024-01-21T03:20:04.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f6aa1840640 10 monclient: init 2024-01-21T03:20:04.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.009+0000 7f189d62e640 1 Processor -- start 2024-01-21T03:20:04.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.009+0000 7f189d62e640 1 -- start start 2024-01-21T03:20:04.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.009+0000 7f189d62e640 10 monclient: init 2024-01-21T03:20:04.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f8275c46640 1 Processor -- start 2024-01-21T03:20:04.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f3eece00640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f3eece00640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-01-21T03:20:04.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f189d62e640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f189d62e640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-01-21T03:20:04.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f189d62e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1890039010 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:04.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f189d62e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1890035670 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:04.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f189d62e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18900cc800 0x7f18900d7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f189d62e640 10 monclient(hunting): picked mon.noname-b con 0x7f18900cc800 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f189d62e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1890035c20 con 0x7f1890035670 2024-01-21T03:20:04.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f189d62e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1890037410 con 0x7f1890039010 2024-01-21T03:20:04.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f189d62e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f189d62e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18900c9d80 con 0x7f18900cc800 2024-01-21T03:20:04.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f189d62e640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f189d62e640 10 monclient(hunting): authenticate will time out at 1350.326706s 2024-01-21T03:20:04.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f3eece00640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3ee80aeb50 0x7f3ee813b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f3eece00640 10 monclient(hunting): picked mon.noname-a con 0x7f3ee80aeb50 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:04.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-01-21T03:20:04.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6a94038db0 0x7f6a940c5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 10 monclient(hunting): picked mon.noname-a con 0x7f6a94038db0 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:04.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a94039190 0x7f6a940c8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 10 monclient(hunting): picked mon.noname-b con 0x7f6a94039190 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6a940c57a0 0x7f6a940c8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f8275c46640 1 -- start start 2024-01-21T03:20:04.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.010+0000 7f8275c46640 10 monclient: init 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f8275c46640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f8275c46640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f8275c46640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8270070640 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f8275c46640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8270071910 0x7f827013cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f8275c46640 10 monclient(hunting): picked mon.noname-c con 0x7f8270071910 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f8275c46640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8270140930 0x7f827013d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f8275c46640 10 monclient(hunting): picked mon.noname-a con 0x7f8270140930 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f8275c46640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827013fd90 con 0x7f8270070640 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f8275c46640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f8275c46640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827013d9a0 con 0x7f8270140930 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f8275c46640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f8275c46640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82700a8a10 con 0x7f8270071910 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f8275c46640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f8275c46640 10 monclient(hunting): authenticate will time out at 1350.327758s 2024-01-21T03:20:04.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 10 monclient(hunting): picked mon.noname-c con 0x7f6a940c57a0 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:04.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a940375b0 con 0x7f6a94038db0 2024-01-21T03:20:04.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a940c7ea0 con 0x7f6a940c57a0 2024-01-21T03:20:04.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940358a0 con 0x7f6a94039190 2024-01-21T03:20:04.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f6aa1840640 10 monclient(hunting): authenticate will time out at 1350.327705s 2024-01-21T03:20:04.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f189ce2d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18900cc800 0x7f18900d7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f1897fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f1890039010 0x7f18900c9670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:44350/0 (socket says 172.21.15.7:44350) 2024-01-21T03:20:04.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f1897fff640 1 -- 172.21.15.7:0/466046802 learned_addr learned my addr 172.21.15.7:0/466046802 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:04.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f3eece00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3ee813b5b0 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:04.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f3eece00640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3ee8072480 0x7f3ee81486e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f3eece00640 10 monclient(hunting): picked mon.noname-b con 0x7f3ee8072480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f3eece00640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee80712b0 con 0x7f3ee813b5b0 2024-01-21T03:20:04.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f3eece00640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f3eece00640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee80aef30 con 0x7f3ee80aeb50 2024-01-21T03:20:04.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f3eece00640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f40ea627640 10 monclient: get_monmap_and_config 2024-01-21T03:20:04.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f40ea627640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.011+0000 7f40ea627640 10 monclient: monmap: 2024-01-21T03:20:04.009 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.009 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.009 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.012402+0000 2024-01-21T03:20:04.009 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.012402+0000 2024-01-21T03:20:04.009 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.009 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.009 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:04.009 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:04.010 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:04.010 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:04.010 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:04.010 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f3eece00640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee8071500 con 0x7f3ee8072480 2024-01-21T03:20:04.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f3eece00640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f3eece00640 10 monclient(hunting): authenticate will time out at 1350.328320s 2024-01-21T03:20:04.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f6a9bfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a94039190 0x7f6a940c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f3edffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3ee813b5b0 0x7f3ee8071d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59826/0 (socket says 172.21.15.7:59826) 2024-01-21T03:20:04.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f3edffff640 1 -- 172.21.15.7:0/3326937985 learned_addr learned my addr 172.21.15.7:0/3326937985 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:04.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f6a9bfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a94039190 0x7f6a940c8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45792/0 (socket says 172.21.15.7:45792) 2024-01-21T03:20:04.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f6a9bfff640 1 -- 172.21.15.7:0/2101539154 learned_addr learned my addr 172.21.15.7:0/2101539154 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f6a9bfff640 10 monclient(hunting): get_auth_request con 0x7f6a94039190 auth_method 0 2024-01-21T03:20:04.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f6a9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f6a9bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f6a9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.012+0000 7f40ea627640 1 Processor -- start 2024-01-21T03:20:04.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.013+0000 7f189ce2d640 10 monclient(hunting): get_auth_request con 0x7f18900cc800 auth_method 0 2024-01-21T03:20:04.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.013+0000 7f189ce2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.013+0000 7f189ce2d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.013+0000 7f189ce2d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.013+0000 7f40ea627640 1 -- start start 2024-01-21T03:20:04.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.013+0000 7f40ea627640 10 monclient: init 2024-01-21T03:20:04.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.013+0000 7f826effd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8270071910 0x7f827013cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.013+0000 7f826effd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8270071910 0x7f827013cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36528/0 (socket says 172.21.15.7:36528) 2024-01-21T03:20:04.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.013+0000 7f826effd640 1 -- 172.21.15.7:0/303179466 learned_addr learned my addr 172.21.15.7:0/303179466 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.013+0000 7f826effd640 10 monclient(hunting): get_auth_request con 0x7f8270071910 auth_method 0 2024-01-21T03:20:04.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.013+0000 7f826effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.013+0000 7f826effd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f826effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.013+0000 7f6aa103f640 1 --2- 172.21.15.7:0/2101539154 >> v2:172.21.15.177:3300/0 conn(0x7f6a940c57a0 0x7f6a940c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f189ce2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f189ce2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f189ce2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-01-21T03:20:04.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40e4070b50 0x7f40e41401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 10 monclient(hunting): picked mon.noname-a con 0x7f40e4070b50 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:04.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f6a9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f6a9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f6a9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f6aa103f640 10 monclient(hunting): get_auth_request con 0x7f6a940c57a0 auth_method 0 2024-01-21T03:20:04.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f6aa103f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f6aa103f640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f6aa103f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40e4071dd0 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:04.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f40e413d5a0 0x7f40e4140720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 10 monclient(hunting): picked mon.noname-b con 0x7f40e413d5a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e40a8960 con 0x7f40e4071dd0 2024-01-21T03:20:04.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e40a8c40 con 0x7f40e4070b50 2024-01-21T03:20:04.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e413fcc0 con 0x7f40e413d5a0 2024-01-21T03:20:04.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.014+0000 7f40ea627640 10 monclient(hunting): authenticate will time out at 1350.330556s 2024-01-21T03:20:04.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.015+0000 7f826effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.015+0000 7f826effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.015+0000 7f826effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f189ce2d640 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2024-01-21T03:20:04.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f189ce2d640 1 -- 172.21.15.7:0/466046802 >> v1:172.21.15.177:6789/0 conn(0x7f1890039010 legacy=0x7f18900c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f189ce2d640 1 -- 172.21.15.7:0/466046802 >> v1:172.21.15.7:6789/0 conn(0x7f1890035670 legacy=0x7f18900cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f189ce2d640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f189ce2d640 1 monclient: found mon.noname-b 2024-01-21T03:20:04.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f189ce2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f189ce2d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f189ce2d640 1 -- 172.21.15.7:0/466046802 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1890037680 con 0x7f18900cc800 2024-01-21T03:20:04.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830689744 0 0) 0x7f3ee80712b0 con 0x7f3ee813b5b0 2024-01-21T03:20:04.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f3ee5d9b640 10 monclient(hunting): _negotiate 2024-01-21T03:20:04.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f3ee5d9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f3ee5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f3ee5d9b640 10 monclient(hunting): my global_id is 4272 2024-01-21T03:20:04.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.016+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed4003520 con 0x7f3ee813b5b0 2024-01-21T03:20:04.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f826effd640 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2024-01-21T03:20:04.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f826effd640 1 -- 172.21.15.7:0/303179466 >> v2:172.21.15.7:3300/0 conn(0x7f8270140930 msgr2=0x7f827013d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f826effd640 1 --2- 172.21.15.7:0/303179466 >> v2:172.21.15.7:3300/0 conn(0x7f8270140930 0x7f827013d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f826effd640 1 -- 172.21.15.7:0/303179466 >> v1:172.21.15.7:6789/0 conn(0x7f8270070640 legacy=0x7f8270140220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f826effd640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f826effd640 1 monclient: found mon.noname-c 2024-01-21T03:20:04.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f826effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f826effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f826effd640 1 -- 172.21.15.7:0/303179466 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82700714c0 con 0x7f8270071910 2024-01-21T03:20:04.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f6a9bfff640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-01-21T03:20:04.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f6a9bfff640 1 -- 172.21.15.7:0/2101539154 >> v2:172.21.15.177:3300/0 conn(0x7f6a940c57a0 msgr2=0x7f6a940c8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f6a9bfff640 1 --2- 172.21.15.7:0/2101539154 >> v2:172.21.15.177:3300/0 conn(0x7f6a940c57a0 0x7f6a940c8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f6a9bfff640 1 -- 172.21.15.7:0/2101539154 >> v2:172.21.15.7:3300/0 conn(0x7f6a94038db0 msgr2=0x7f6a940c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f6a9bfff640 1 --2- 172.21.15.7:0/2101539154 >> v2:172.21.15.7:3300/0 conn(0x7f6a94038db0 0x7f6a940c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f6a9bfff640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f6a9bfff640 1 monclient: found mon.noname-b 2024-01-21T03:20:04.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f6a9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f6a9bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.017+0000 7f6a9bfff640 1 -- 172.21.15.7:0/2101539154 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a940360a0 con 0x7f6a94039190 2024-01-21T03:20:04.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.020+0000 7fc3c5475640 10 monclient: get_monmap_and_config 2024-01-21T03:20:04.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.020+0000 7fc3c5475640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.020+0000 7fc3c5475640 10 monclient: monmap: 2024-01-21T03:20:04.024 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.024 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.024 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.021038+0000 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.021038+0000 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.020+0000 7fc3c5475640 1 Processor -- start 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.021+0000 7fc3c5475640 1 -- start start 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.021+0000 7fc3c5475640 10 monclient: init 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.021+0000 7fc3c5475640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.021+0000 7fc3c5475640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-01-21T03:20:04.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.021+0000 7fc3c5475640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc3b8038de0 0x7fc3b80c5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.021+0000 7fc3c5475640 10 monclient(hunting): picked mon.noname-a con 0x7fc3b8038de0 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:04.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.021+0000 7fc3c5475640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc3b80391c0 0x7fc3b80c8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.021+0000 7fc3c5475640 10 monclient(hunting): picked mon.noname-c con 0x7fc3b80391c0 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:04.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.022+0000 7fc3c5475640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3b80c57d0 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:04.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.022+0000 7fc3c5475640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b80c7ed0 con 0x7fc3b80c57d0 2024-01-21T03:20:04.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.022+0000 7fc3c5475640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.022+0000 7fc3c5475640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b8037670 con 0x7fc3b8038de0 2024-01-21T03:20:04.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.022+0000 7fc3c5475640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.022+0000 7fc3c5475640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b80354a0 con 0x7fc3b80391c0 2024-01-21T03:20:04.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.022+0000 7fc3c5475640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.022+0000 7fc3c5475640 10 monclient(hunting): authenticate will time out at 1350.337884s 2024-01-21T03:20:04.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.022+0000 7f40e8e24640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f40e4071dd0 0x7f40e413ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:44356/0 (socket says 172.21.15.7:44356) 2024-01-21T03:20:04.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.022+0000 7f40e8e24640 1 -- 172.21.15.7:0/2426861665 learned_addr learned my addr 172.21.15.7:0/2426861665 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:04.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.022+0000 7f6a9bfff640 1 --2- 172.21.15.7:0/2101539154 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a94039190 0x7f6a940c8350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6a88009870 tx=0x7f6a8802b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8443308c9d777de server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.022+0000 7f826effd640 1 --2- 172.21.15.7:0/303179466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8270071910 0x7f827013cf20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f82700b63e0 tx=0x7f826402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2844548f9b068c95 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.023+0000 7f189ce2d640 1 --2- 172.21.15.7:0/466046802 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18900cc800 0x7f18900d7970 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f188c0097d0 tx=0x7f188c02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b970a2c1bddb1d7c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.023+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 756906883 0 0) 0x7f3ed4003520 con 0x7f3ee813b5b0 2024-01-21T03:20:04.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.023+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3ee8072480 msgr2=0x7f3ee81486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.023+0000 7f3ee5d9b640 1 --2- 172.21.15.7:0/3326937985 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3ee8072480 0x7f3ee81486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.023+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 >> v2:172.21.15.7:3300/0 conn(0x7f3ee80aeb50 msgr2=0x7f3ee813b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.023+0000 7f3ee5d9b640 1 --2- 172.21.15.7:0/3326937985 >> v2:172.21.15.7:3300/0 conn(0x7f3ee80aeb50 0x7f3ee813b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.023+0000 7f3ee5d9b640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.023+0000 7f3ee5d9b640 1 monclient: found mon.noname-a-legacy 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.023+0000 7f3ee5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.023+0000 7f3ee5d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.7:6789/0 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.023+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee80717e0 con 0x7f3ee813b5b0 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 749591385 0 0) 0x7f3ed0001670 con 0x7f3ee813b5b0 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3eece00640 5 monclient: authenticate success, global_id 4272 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3eece00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:04.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient: dump: 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] but i'm connected to v1:172.21.15.7:6789/0, reconnecting 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 >> v1:172.21.15.7:6789/0 conn(0x7f3ee813b5b0 legacy=0x7f3ee8071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 1 --2- 172.21.15.7:0/3326937985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ed4003970 0x7f3ed4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient(hunting): picked mon.a con 0x7f3ed4003970 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 1 --2- 172.21.15.7:0/3326937985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed40047c0 0x7f3ed4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient(hunting): picked mon.b con 0x7f3ed40047c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 1 --2- 172.21.15.7:0/3326937985 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3ee8072480 0x7f3ed4005950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient(hunting): picked mon.c con 0x7f3ee8072480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80aef30 con 0x7f3ed4003970 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee8071500 con 0x7f3ed40047c0 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed4005540 con 0x7f3ee8072480 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3ee5d9b640 10 monclient(hunting): _finish_auth 0 2024-01-21T03:20:04.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.024+0000 7f3eece00640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:04.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.025+0000 7fc3be7fc640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc3b80391c0 0x7fc3b80c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.025+0000 7f50349e2640 10 monclient: get_monmap_and_config 2024-01-21T03:20:04.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.025+0000 7f50349e2640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.026+0000 7f50349e2640 10 monclient: monmap: 2024-01-21T03:20:04.032 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.026977+0000 2024-01-21T03:20:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.026977+0000 2024-01-21T03:20:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:04.033 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.027+0000 7f6a9b7fe640 1 -- 172.21.15.7:0/2101539154 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a880317e0 con 0x7f6a94039190 2024-01-21T03:20:04.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.027+0000 7f6a9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.027+0000 7f6a9b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:04.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.027+0000 7f50349e2640 1 Processor -- start 2024-01-21T03:20:04.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.027+0000 7f6a9b7fe640 10 monclient: dump: 2024-01-21T03:20:04.034 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.034 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.034 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.035 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.027+0000 7f6a9b7fe640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.027+0000 7f6a9b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.027+0000 7f6a9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.027+0000 7f6aa1840640 5 monclient: authenticate success, global_id 4262 2024-01-21T03:20:04.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.027+0000 7f6aa1840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:04.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.027+0000 7f50349e2640 1 -- start start 2024-01-21T03:20:04.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.027+0000 7f50349e2640 10 monclient: init 2024-01-21T03:20:04.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f826e7fc640 1 -- 172.21.15.7:0/303179466 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264031870 con 0x7f8270071910 2024-01-21T03:20:04.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f826e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f826e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:04.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f826e7fc640 10 monclient: dump: 2024-01-21T03:20:04.036 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f826e7fc640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f826e7fc640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f826e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f8275c46640 5 monclient: authenticate success, global_id 4288 2024-01-21T03:20:04.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f8275c46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:04.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-01-21T03:20:04.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5030070a10 0x7f5030140380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 10 monclient(hunting): picked mon.noname-b con 0x7f5030070a10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5030070df0 0x7f50301408c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5030070df0 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:04.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5030143400 0x7f5030143840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5030143400 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5030140e00 con 0x7f5030070df0 2024-01-21T03:20:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f503013fe80 con 0x7f5030143400 2024-01-21T03:20:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50300a8610 con 0x7f5030070a10 2024-01-21T03:20:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.028+0000 7f50349e2640 10 monclient(hunting): authenticate will time out at 1350.344522s 2024-01-21T03:20:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.029+0000 7fc3be7fc640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc3b80391c0 0x7fc3b80c8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36544/0 (socket says 172.21.15.7:36544) 2024-01-21T03:20:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.029+0000 7fc3be7fc640 1 -- 172.21.15.7:0/2300227020 learned_addr learned my addr 172.21.15.7:0/2300227020 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.029+0000 7fc3be7fc640 10 monclient(hunting): get_auth_request con 0x7fc3b80391c0 auth_method 0 2024-01-21T03:20:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.029+0000 7fc3be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.029+0000 7fc3be7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.029+0000 7fc3be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f826e7fc640 1 -- 172.21.15.7:0/303179466 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82640319d0 con 0x7f8270071910 2024-01-21T03:20:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f826e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f8275c46640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f8275c46640 4 set_mon_vals no callback set 2024-01-21T03:20:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f826e7fc640 1 -- 172.21.15.7:0/303179466 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264031d60 con 0x7f8270071910 2024-01-21T03:20:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f826e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f826e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f826e7fc640 10 monclient: dump: 2024-01-21T03:20:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.043 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7fc3beffd640 1 --2- 172.21.15.7:0/2300227020 >> v2:172.21.15.7:3300/0 conn(0x7fc3b8038de0 0x7fc3b80c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f00ff7e2640 10 monclient: get_monmap_and_config 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f00ff7e2640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f00ff7e2640 10 monclient: monmap: 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.031675+0000 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.031675+0000 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:04.044 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7fc3beffd640 10 monclient(hunting): get_auth_request con 0x7fc3b8038de0 auth_method 0 2024-01-21T03:20:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7fc3beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7fc3beffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7fc3beffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f8275c46640 10 monclient: shutdown 2024-01-21T03:20:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f8275c46640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f8275c46640 1 -- 172.21.15.7:0/303179466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8270071910 msgr2=0x7f827013cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f8275c46640 1 --2- 172.21.15.7:0/303179466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8270071910 0x7f827013cf20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f82700b63e0 tx=0x7f826402b910 comp rx=0 tx=0).stop 2024-01-21T03:20:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f8275c46640 1 -- 172.21.15.7:0/303179466 shutdown_connections 2024-01-21T03:20:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.030+0000 7f8275c46640 1 --2- 172.21.15.7:0/303179466 >> v2:172.21.15.7:3300/0 conn(0x7f8270140930 0x7f827013d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f8275c46640 1 --2- 172.21.15.7:0/303179466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8270071910 0x7f827013cf20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f8275c46640 1 -- 172.21.15.7:0/303179466 >> 172.21.15.7:0/303179466 conn(0x7f8270066d30 msgr2=0x7f82700b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f8275c46640 1 -- 172.21.15.7:0/303179466 shutdown_connections 2024-01-21T03:20:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f8275c46640 1 -- 172.21.15.7:0/303179466 wait complete. 2024-01-21T03:20:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7fc3be7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7fc3be7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7fc3be7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f40dbfff640 1 -- 172.21.15.7:0/2426861665 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720012984 0 0) 0x7f40e40a8960 con 0x7f40e4071dd0 2024-01-21T03:20:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f502ed9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5030143400 0x7f5030143840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f00ff7e2640 1 Processor -- start 2024-01-21T03:20:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f00ff7e2640 1 -- start start 2024-01-21T03:20:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f6a9b7fe640 1 -- 172.21.15.7:0/2101539154 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a88031940 con 0x7f6a94039190 2024-01-21T03:20:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f00ff7e2640 10 monclient: init 2024-01-21T03:20:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f6a9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f40dbfff640 10 monclient(hunting): _negotiate 2024-01-21T03:20:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f40dbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f40dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f40dbfff640 10 monclient(hunting): my global_id is 4297 2024-01-21T03:20:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f40dbfff640 1 -- 172.21.15.7:0/2426861665 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d0003520 con 0x7f40e4071dd0 2024-01-21T03:20:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f3ee659c640 1 --2- 172.21.15.7:0/3326937985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ed4003970 0x7f3ed4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f6aa1840640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f6aa1840640 4 set_mon_vals no callback set 2024-01-21T03:20:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f6a9b7fe640 1 -- 172.21.15.7:0/2101539154 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a88031cd0 con 0x7f6a94039190 2024-01-21T03:20:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f6a9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f6a9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.031+0000 7f6a9b7fe640 10 monclient: dump: 2024-01-21T03:20:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.051 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.051 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.051 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.051 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.051 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.051 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.051 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f3edffff640 1 --2- 172.21.15.7:0/3326937985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed40047c0 0x7f3ed4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f8275c46640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f8275c46640 10 monclient: monmap: 2024-01-21T03:20:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.052 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f502ed9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5030143400 0x7f5030143840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36552/0 (socket says 172.21.15.7:36552) 2024-01-21T03:20:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f502ed9d640 1 -- 172.21.15.7:0/1231665372 learned_addr learned my addr 172.21.15.7:0/1231665372 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f502ed9d640 10 monclient(hunting): get_auth_request con 0x7f5030143400 auth_method 0 2024-01-21T03:20:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f502ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f502ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f502ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f3ee6d9d640 1 --2- 172.21.15.7:0/3326937985 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3ee8072480 0x7f3ed4005950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f3edffff640 10 monclient(hunting): get_auth_request con 0x7f3ed40047c0 auth_method 0 2024-01-21T03:20:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f3edffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f3edffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f3edffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:20:04.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f6aa1840640 10 monclient: shutdown 2024-01-21T03:20:04.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-01-21T03:20:04.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f6aa1840640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f6aa1840640 1 -- 172.21.15.7:0/2101539154 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a94039190 msgr2=0x7f6a940c8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f6aa1840640 1 --2- 172.21.15.7:0/2101539154 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a94039190 0x7f6a940c8350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6a88009870 tx=0x7f6a8802b880 comp rx=0 tx=0).stop 2024-01-21T03:20:04.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00f80aed70 0x7f00f80af150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f8275c46640 1 librados: starting msgr at 2024-01-21T03:20:04.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f8275c46640 1 librados: starting objecter 2024-01-21T03:20:04.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f8275c46640 1 Processor -- start 2024-01-21T03:20:04.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 10 monclient(hunting): picked mon.noname-b con 0x7f00f80aed70 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f00f80afff0 0x7f00f8142640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f6aa1840640 1 -- 172.21.15.7:0/2101539154 shutdown_connections 2024-01-21T03:20:04.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f6aa1840640 1 --2- 172.21.15.7:0/2101539154 >> v2:172.21.15.177:3300/0 conn(0x7f6a940c57a0 0x7f6a940c8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f6aa1840640 1 --2- 172.21.15.7:0/2101539154 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a94039190 0x7f6a940c8350 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f6aa1840640 1 --2- 172.21.15.7:0/2101539154 >> v2:172.21.15.7:3300/0 conn(0x7f6a94038db0 0x7f6a940c5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 10 monclient(hunting): picked mon.noname-c con 0x7f00f80afff0 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:04.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f6aa1840640 1 -- 172.21.15.7:0/2101539154 >> 172.21.15.7:0/2101539154 conn(0x7f6a94006610 msgr2=0x7f6a94006a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f00f8142b80 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:04.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f806f460 con 0x7f00f8142b80 2024-01-21T03:20:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f80afba0 con 0x7f00f80afff0 2024-01-21T03:20:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8142120 con 0x7f00f80aed70 2024-01-21T03:20:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f00ff7e2640 10 monclient(hunting): authenticate will time out at 1350.348552s 2024-01-21T03:20:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f6aa1840640 1 -- 172.21.15.7:0/2101539154 shutdown_connections 2024-01-21T03:20:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f6aa1840640 1 -- 172.21.15.7:0/2101539154 wait complete. 2024-01-21T03:20:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f8275c46640 1 -- start start 2024-01-21T03:20:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f8275c46640 1 librados: setting wanted keys 2024-01-21T03:20:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f8275c46640 1 librados: calling monclient init 2024-01-21T03:20:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.032+0000 7f8275c46640 10 monclient: init 2024-01-21T03:20:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f40e9625640 1 --2- 172.21.15.7:0/2426861665 >> v2:172.21.15.7:3300/0 conn(0x7f40e4070b50 0x7f40e41401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f3ee6d9d640 10 monclient(hunting): get_auth_request con 0x7f3ee8072480 auth_method 0 2024-01-21T03:20:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f3ee6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f3ee6d9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f3ee6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:20:04.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f6aa1840640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f6aa1840640 10 monclient: monmap: 2024-01-21T03:20:04.060 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.060 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.060 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.060 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.060 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.060 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.060 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.060 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.061 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.061 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:20:04.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8270070640 0x7f82700a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 10 monclient(hunting): picked mon.noname-a con 0x7f8270070640 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8270071910 0x7f82700ac190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 10 monclient(hunting): picked mon.noname-c con 0x7f8270071910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8270140930 0x7f82700aa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 10 monclient(hunting): picked mon.noname-b con 0x7f8270140930 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82700a9fc0 con 0x7f8270070640 2024-01-21T03:20:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82700ab7b0 con 0x7f8270140930 2024-01-21T03:20:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.033+0000 7f8275c46640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82700aba20 con 0x7f8270071910 2024-01-21T03:20:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f8275c46640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f8275c46640 10 monclient(hunting): authenticate will time out at 1350.349778s 2024-01-21T03:20:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 1 librados: starting msgr at 2024-01-21T03:20:04.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 1 librados: starting objecter 2024-01-21T03:20:04.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f502e59c640 1 --2- 172.21.15.7:0/1231665372 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5030070a10 0x7f5030140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 1 Processor -- start 2024-01-21T03:20:04.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 1 -- start start 2024-01-21T03:20:04.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 1 librados: setting wanted keys 2024-01-21T03:20:04.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 1 librados: calling monclient init 2024-01-21T03:20:04.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 10 monclient: init 2024-01-21T03:20:04.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f3ee6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f3ee6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f3ee6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T03:20:04.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f3edffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f3edffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f3edffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T03:20:04.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T03:20:04.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a94038db0 0x7f6a940c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 10 monclient(hunting): picked mon.noname-c con 0x7f6a94038db0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a940c57a0 0x7f6a94032500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 10 monclient(hunting): picked mon.noname-b con 0x7f6a940c57a0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.034+0000 7f6aa1840640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a940ca590 0x7f6a94032a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.035+0000 7f6aa1840640 10 monclient(hunting): picked mon.noname-a con 0x7f6a940ca590 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.035+0000 7f6aa1840640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.035+0000 7f6aa1840640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a94031e00 con 0x7f6a940ca590 2024-01-21T03:20:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.035+0000 7f6aa1840640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.035+0000 7f6aa1840640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940320e0 con 0x7f6a940c57a0 2024-01-21T03:20:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.035+0000 7f6aa1840640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.035+0000 7f6aa1840640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940caa20 con 0x7f6a94038db0 2024-01-21T03:20:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.035+0000 7f502ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.035+0000 7f502ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.035+0000 7f502ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.035+0000 7f6aa1840640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.035+0000 7f6aa1840640 10 monclient(hunting): authenticate will time out at 1350.350871s 2024-01-21T03:20:04.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f1896ffd640 1 -- 172.21.15.7:0/466046802 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c031770 con 0x7f18900cc800 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f1896ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f1896ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f1896ffd640 10 monclient: dump: 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f1896ffd640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f1896ffd640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f1896ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f1896ffd640 1 -- 172.21.15.7:0/466046802 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188c0318d0 con 0x7f18900cc800 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f1896ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f1896ffd640 1 -- 172.21.15.7:0/466046802 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c031c60 con 0x7f18900cc800 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f189d62e640 5 monclient: authenticate success, global_id 4259 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f189d62e640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f1896ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f189d62e640 4 set_mon_vals no callback set 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f1896ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f1896ffd640 10 monclient: dump: 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.070 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.071 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.071 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.071 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.071 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.071 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f502e59c640 10 monclient(hunting): get_auth_request con 0x7f5030070a10 auth_method 0 2024-01-21T03:20:04.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f502e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f502e59c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f502e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.036+0000 7f40e9e26640 1 --2- 172.21.15.7:0/2426861665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f40e413d5a0 0x7f40e4140720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f189d62e640 10 monclient: shutdown 2024-01-21T03:20:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f189d62e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f189d62e640 1 -- 172.21.15.7:0/466046802 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18900cc800 msgr2=0x7f18900d7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f189d62e640 1 --2- 172.21.15.7:0/466046802 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18900cc800 0x7f18900d7970 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f188c0097d0 tx=0x7f188c02b810 comp rx=0 tx=0).stop 2024-01-21T03:20:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f3edffff640 10 monclient(hunting): handle_auth_done global_id 4272 payload 293 2024-01-21T03:20:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f3edffff640 1 -- 172.21.15.7:0/3326937985 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3ee8072480 msgr2=0x7f3ed4005950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f3edffff640 1 --2- 172.21.15.7:0/3326937985 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3ee8072480 0x7f3ed4005950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f189d62e640 1 -- 172.21.15.7:0/466046802 shutdown_connections 2024-01-21T03:20:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f189d62e640 1 --2- 172.21.15.7:0/466046802 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18900cc800 0x7f18900d7970 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f189d62e640 1 -- 172.21.15.7:0/466046802 >> 172.21.15.7:0/466046802 conn(0x7f1890006600 msgr2=0x7f1890006a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f3edffff640 1 -- 172.21.15.7:0/3326937985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ed4003970 msgr2=0x7f3ed4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f3edffff640 1 --2- 172.21.15.7:0/3326937985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ed4003970 0x7f3ed4003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f3edffff640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f3edffff640 1 monclient: found mon.b 2024-01-21T03:20:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f3edffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f3edffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f3edffff640 1 -- 172.21.15.7:0/3326937985 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed4003520 con 0x7f3ed40047c0 2024-01-21T03:20:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f3edffff640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f3edffff640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f3edffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f189d62e640 1 -- 172.21.15.7:0/466046802 shutdown_connections 2024-01-21T03:20:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f189d62e640 1 -- 172.21.15.7:0/466046802 wait complete. 2024-01-21T03:20:04.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f40e9625640 10 monclient(hunting): get_auth_request con 0x7f40e4070b50 auth_method 0 2024-01-21T03:20:04.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f40e9625640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f40e9625640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.037+0000 7f40e9625640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.038+0000 7f189d62e640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.038+0000 7f189d62e640 10 monclient: monmap: 2024-01-21T03:20:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.076 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.038+0000 7f189d62e640 1 librados: starting msgr at 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.038+0000 7f189d62e640 1 librados: starting objecter 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.038+0000 7f189d62e640 1 Processor -- start 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.038+0000 7f189d62e640 1 -- start start 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.038+0000 7f189d62e640 1 librados: setting wanted keys 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.038+0000 7f189d62e640 1 librados: calling monclient init 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.038+0000 7f189d62e640 10 monclient: init 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1890035670 0x7f1890033c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 10 monclient(hunting): picked mon.noname-b con 0x7f1890035670 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1890039010 0x7f18900bbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 10 monclient(hunting): picked mon.noname-a con 0x7f1890039010 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18900cc800 0x7f18900313f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 10 monclient(hunting): picked mon.noname-c con 0x7f18900cc800 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18900be310 con 0x7f1890039010 2024-01-21T03:20:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18900bd620 con 0x7f1890035670 2024-01-21T03:20:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1890032920 con 0x7f18900cc800 2024-01-21T03:20:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.039+0000 7f189d62e640 10 monclient(hunting): authenticate will time out at 1350.355116s 2024-01-21T03:20:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f502ed9d640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2024-01-21T03:20:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f502ed9d640 1 -- 172.21.15.7:0/1231665372 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5030070a10 msgr2=0x7f5030140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f502ed9d640 1 --2- 172.21.15.7:0/1231665372 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5030070a10 0x7f5030140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f502ed9d640 1 -- 172.21.15.7:0/1231665372 >> v2:172.21.15.7:3300/0 conn(0x7f5030070df0 msgr2=0x7f50301408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f502ed9d640 1 --2- 172.21.15.7:0/1231665372 >> v2:172.21.15.7:3300/0 conn(0x7f5030070df0 0x7f50301408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f502ed9d640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f502ed9d640 1 monclient: found mon.noname-c 2024-01-21T03:20:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f502ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f502ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f502ed9d640 1 -- 172.21.15.7:0/1231665372 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50300a88f0 con 0x7f5030143400 2024-01-21T03:20:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f40e9e26640 10 monclient(hunting): get_auth_request con 0x7f40e413d5a0 auth_method 0 2024-01-21T03:20:04.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f40e9e26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f40e9e26640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.040+0000 7f40e9e26640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.041+0000 7fc3beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.041+0000 7fc3beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.041+0000 7fc3beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.042+0000 7fc3bdffb640 1 -- 172.21.15.7:0/2300227020 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763491156 0 0) 0x7fc3b80c7ed0 con 0x7fc3b80c57d0 2024-01-21T03:20:04.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.042+0000 7fc3bdffb640 10 monclient(hunting): _negotiate 2024-01-21T03:20:04.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.042+0000 7fc3bdffb640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.042+0000 7fc3bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.042+0000 7fc3bdffb640 10 monclient(hunting): my global_id is 4306 2024-01-21T03:20:04.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.042+0000 7fc3bdffb640 1 -- 172.21.15.7:0/2300227020 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a8003520 con 0x7fc3b80c57d0 2024-01-21T03:20:04.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.042+0000 7f40e9e26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.042+0000 7f40e9e26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.042+0000 7f40e9e26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.042+0000 7f3edffff640 1 --2- 172.21.15.7:0/3326937985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed40047c0 0x7f3ed4004c40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0007c00 tx=0x7f3ed40057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f8ed684371031d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.042+0000 7f40e9625640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.042+0000 7f40e9625640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.042+0000 7f40e9625640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.044+0000 7f40e9625640 10 monclient(hunting): handle_auth_done global_id 4278 payload 274 2024-01-21T03:20:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.044+0000 7f40e9625640 1 -- 172.21.15.7:0/2426861665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f40e413d5a0 msgr2=0x7f40e4140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.044+0000 7f40e9625640 1 --2- 172.21.15.7:0/2426861665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f40e413d5a0 0x7f40e4140720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.044+0000 7f40e9625640 1 -- 172.21.15.7:0/2426861665 >> v1:172.21.15.177:6789/0 conn(0x7f40e4071dd0 legacy=0x7f40e413ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.044+0000 7f40e9625640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.044+0000 7f40e9625640 1 monclient: found mon.noname-a 2024-01-21T03:20:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.044+0000 7f40e9625640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.044+0000 7f40e9625640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.044+0000 7f40e9625640 1 -- 172.21.15.7:0/2426861665 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e40ae1a0 con 0x7f40e4070b50 2024-01-21T03:20:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.047+0000 7fc3beffd640 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2024-01-21T03:20:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.047+0000 7fc3beffd640 1 -- 172.21.15.7:0/2300227020 >> v2:172.21.15.177:3300/0 conn(0x7fc3b80391c0 msgr2=0x7fc3b80c8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.047+0000 7fc3beffd640 1 --2- 172.21.15.7:0/2300227020 >> v2:172.21.15.177:3300/0 conn(0x7fc3b80391c0 0x7fc3b80c8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7fc3beffd640 1 -- 172.21.15.7:0/2300227020 >> v1:172.21.15.177:6789/0 conn(0x7fc3b80c57d0 legacy=0x7fc3b80cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7fc3beffd640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7fc3beffd640 1 monclient: found mon.noname-a 2024-01-21T03:20:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7fc3beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7fc3beffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7fc3beffd640 1 -- 172.21.15.7:0/2300227020 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b80358d0 con 0x7fc3b8038de0 2024-01-21T03:20:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7f00fd57e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00f80aed70 0x7f00f80af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7f502ed9d640 1 --2- 172.21.15.7:0/1231665372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5030143400 0x7f5030143840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f50300afaf0 tx=0x7f502802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a762c8ceef52b1b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7f00fd57e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00f80aed70 0x7f00f80af150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45832/0 (socket says 172.21.15.7:45832) 2024-01-21T03:20:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7f00fd57e640 1 -- 172.21.15.7:0/1842429924 learned_addr learned my addr 172.21.15.7:0/1842429924 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7f00fd57e640 10 monclient(hunting): get_auth_request con 0x7f00f80aed70 auth_method 0 2024-01-21T03:20:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7f00fd57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7f00fd57e640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7f00fd57e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.048+0000 7f826effd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8270071910 0x7f82700ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.049+0000 7f40e9625640 1 --2- 172.21.15.7:0/2426861665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4070b50 0x7f40e41401e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f40d40097a0 tx=0x7f40d402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0bba7cb1d2f110 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.050+0000 7f00fd57e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.050+0000 7f00fd57e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.050+0000 7f00fd57e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.050+0000 7f826effd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8270071910 0x7f82700ac190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45848/0 (socket says 172.21.15.7:45848) 2024-01-21T03:20:04.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.050+0000 7f826effd640 1 -- 172.21.15.7:0/2865264600 learned_addr learned my addr 172.21.15.7:0/2865264600 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.050+0000 7f826effd640 10 monclient(hunting): get_auth_request con 0x7f8270071910 auth_method 0 2024-01-21T03:20:04.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.050+0000 7f826effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.050+0000 7f826effd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.050+0000 7f826effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.051+0000 7f00fcd7d640 1 --2- 172.21.15.7:0/1842429924 >> v2:172.21.15.177:3300/0 conn(0x7f00f80afff0 0x7f00f8142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.051+0000 7f00fcd7d640 10 monclient(hunting): get_auth_request con 0x7f00f80afff0 auth_method 0 2024-01-21T03:20:04.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.051+0000 7f00fcd7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.051+0000 7f00fcd7d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.051+0000 7f00fcd7d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.052+0000 7f00fd57e640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-01-21T03:20:04.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.052+0000 7f00fd57e640 1 -- 172.21.15.7:0/1842429924 >> v2:172.21.15.177:3300/0 conn(0x7f00f80afff0 msgr2=0x7f00f8142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.052+0000 7f00fd57e640 1 --2- 172.21.15.7:0/1842429924 >> v2:172.21.15.177:3300/0 conn(0x7f00f80afff0 0x7f00f8142640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.052+0000 7f00fd57e640 1 -- 172.21.15.7:0/1842429924 >> v1:172.21.15.177:6789/0 conn(0x7f00f8142b80 legacy=0x7f00f813f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.052+0000 7f00fd57e640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.052+0000 7f00fd57e640 1 monclient: found mon.noname-b 2024-01-21T03:20:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.052+0000 7f00fd57e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.052+0000 7f00fd57e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.052+0000 7f00fd57e640 1 -- 172.21.15.7:0/1842429924 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f80ac8f0 con 0x7f00f80aed70 2024-01-21T03:20:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.053+0000 7f6aa083e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a94038db0 0x7f6a940c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.054+0000 7f826effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.054+0000 7f826effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.054+0000 7f826effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.055+0000 7f6aa083e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a94038db0 0x7f6a940c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45858/0 (socket says 172.21.15.7:45858) 2024-01-21T03:20:04.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.055+0000 7f6aa083e640 1 -- 172.21.15.7:0/2784924632 learned_addr learned my addr 172.21.15.7:0/2784924632 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.055+0000 7f6aa083e640 10 monclient(hunting): get_auth_request con 0x7f6a94038db0 auth_method 0 2024-01-21T03:20:04.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.055+0000 7f6aa083e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.055+0000 7f6aa083e640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.055+0000 7f6aa083e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f502dd9b640 1 -- 172.21.15.7:0/1231665372 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502802ca70 con 0x7f5030143400 2024-01-21T03:20:04.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f502dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f1897fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1890035670 0x7f1890033c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f502dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f502dd9b640 10 monclient: dump: 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f502dd9b640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f502dd9b640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f502dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f502dd9b640 1 -- 172.21.15.7:0/1231665372 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502802cbd0 con 0x7f5030143400 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f50349e2640 5 monclient: authenticate success, global_id 4303 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f50349e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f502dd9b640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f502dd9b640 1 -- 172.21.15.7:0/1231665372 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50280343f0 con 0x7f5030143400 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f50349e2640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.056+0000 7f50349e2640 4 set_mon_vals no callback set 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f502dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f502dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f502dd9b640 10 monclient: dump: 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.095 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f1897fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1890035670 0x7f1890033c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36602/0 (socket says 172.21.15.7:36602) 2024-01-21T03:20:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f1897fff640 1 -- 172.21.15.7:0/488701857 learned_addr learned my addr 172.21.15.7:0/488701857 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f1897fff640 10 monclient(hunting): get_auth_request con 0x7f1890035670 auth_method 0 2024-01-21T03:20:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f1897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f1897fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f1897fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7fc3beffd640 1 --2- 172.21.15.7:0/2300227020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b8038de0 0x7fc3b80c5290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00097a0 tx=0x7fc3b002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d384cd781b2a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f50349e2640 10 monclient: shutdown 2024-01-21T03:20:04.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f50349e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f50349e2640 1 -- 172.21.15.7:0/1231665372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5030143400 msgr2=0x7f5030143840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f50349e2640 1 --2- 172.21.15.7:0/1231665372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5030143400 0x7f5030143840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f50300afaf0 tx=0x7f502802bed0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.057+0000 7f50349e2640 1 -- 172.21.15.7:0/1231665372 shutdown_connections 2024-01-21T03:20:04.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.058+0000 7f50349e2640 1 --2- 172.21.15.7:0/1231665372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5030143400 0x7f5030143840 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.058+0000 7f50349e2640 1 --2- 172.21.15.7:0/1231665372 >> v2:172.21.15.7:3300/0 conn(0x7f5030070df0 0x7f50301408c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.058+0000 7f50349e2640 1 --2- 172.21.15.7:0/1231665372 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5030070a10 0x7f5030140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.058+0000 7f50349e2640 1 -- 172.21.15.7:0/1231665372 >> 172.21.15.7:0/1231665372 conn(0x7f5030066f50 msgr2=0x7f50300b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.058+0000 7f50349e2640 1 -- 172.21.15.7:0/1231665372 shutdown_connections 2024-01-21T03:20:04.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.058+0000 7f50349e2640 1 -- 172.21.15.7:0/1231665372 wait complete. 2024-01-21T03:20:04.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.058+0000 7f1897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.058+0000 7f1897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.058+0000 7f1897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.059+0000 7f50349e2640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.059+0000 7f826effd640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-01-21T03:20:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.059+0000 7f50349e2640 10 monclient: monmap: 2024-01-21T03:20:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.099 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.059+0000 7f826effd640 1 -- 172.21.15.7:0/2865264600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8270140930 msgr2=0x7f82700aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.059+0000 7f826effd640 1 --2- 172.21.15.7:0/2865264600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8270140930 0x7f82700aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.059+0000 7f826effd640 1 -- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8270070640 msgr2=0x7f82700a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.059+0000 7f826effd640 1 --2- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8270070640 0x7f82700a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.059+0000 7f826effd640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.059+0000 7f826effd640 1 monclient: found mon.noname-c 2024-01-21T03:20:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.059+0000 7f826effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.059+0000 7f826effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.059+0000 7f826effd640 1 -- 172.21.15.7:0/2865264600 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82640094d0 con 0x7f8270071910 2024-01-21T03:20:04.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.060+0000 7f50349e2640 1 librados: starting msgr at 2024-01-21T03:20:04.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.060+0000 7f50349e2640 1 librados: starting objecter 2024-01-21T03:20:04.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.060+0000 7f50349e2640 1 Processor -- start 2024-01-21T03:20:04.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.060+0000 7f50349e2640 1 -- start start 2024-01-21T03:20:04.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.060+0000 7f50349e2640 1 librados: setting wanted keys 2024-01-21T03:20:04.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.060+0000 7f50349e2640 1 librados: calling monclient init 2024-01-21T03:20:04.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.060+0000 7f50349e2640 10 monclient: init 2024-01-21T03:20:04.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.060+0000 7f6aa083e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.060+0000 7f6aa083e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.060+0000 7f6aa083e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5030070a10 0x7f50301d8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5030070a10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5030070df0 0x7f5030136a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5030070df0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5030143400 0x7f5030136fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 10 monclient(hunting): picked mon.noname-b con 0x7f5030143400 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50300ac1c0 con 0x7f5030070a10 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5030141530 con 0x7f5030143400 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50300a9d60 con 0x7f5030070df0 2024-01-21T03:20:04.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.061+0000 7f50349e2640 10 monclient(hunting): authenticate will time out at 1350.376979s 2024-01-21T03:20:04.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.062+0000 7f00fd57e640 1 --2- 172.21.15.7:0/1842429924 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00f80aed70 0x7f00f80af150 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f00f4009870 tx=0x7f00f402b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8adfcc86aa5c4c5f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.063+0000 7f826effd640 1 --2- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8270071910 0x7f82700ac190 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f826402b950 tx=0x7f82640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.063+0000 7f8257fff640 10 client.?.objecter ms_handle_connect 0x7f8270071910 2024-01-21T03:20:04.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.064+0000 7f8257fff640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.064+0000 7f1897fff640 10 monclient(hunting): handle_auth_done global_id 4312 payload 824 2024-01-21T03:20:04.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.064+0000 7f1897fff640 1 -- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18900cc800 msgr2=0x7f18900313f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.064+0000 7f1897fff640 1 --2- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18900cc800 0x7f18900313f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.064+0000 7f1897fff640 1 -- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1890039010 msgr2=0x7f18900bbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.064+0000 7f1897fff640 1 --2- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1890039010 0x7f18900bbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.064+0000 7f1897fff640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.064+0000 7f1897fff640 1 monclient: found mon.noname-b 2024-01-21T03:20:04.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.064+0000 7f1897fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.064+0000 7f1897fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.064+0000 7f1897fff640 1 -- 172.21.15.7:0/488701857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188c009400 con 0x7f1890035670 2024-01-21T03:20:04.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f6aa083e640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-01-21T03:20:04.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f6aa083e640 1 -- 172.21.15.7:0/2784924632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a940c57a0 msgr2=0x7f6a94032500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f6aa083e640 1 --2- 172.21.15.7:0/2784924632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a940c57a0 0x7f6a94032500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f6aa083e640 1 -- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a940ca590 msgr2=0x7f6a94032a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f6aa083e640 1 --2- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a940ca590 0x7f6a94032a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f6aa083e640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f6aa083e640 1 monclient: found mon.noname-c 2024-01-21T03:20:04.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f6aa083e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f6aa083e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f6aa083e640 1 -- 172.21.15.7:0/2784924632 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a880094d0 con 0x7f6a94038db0 2024-01-21T03:20:04.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f40dbfff640 1 -- 172.21.15.7:0/2426861665 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d402f050 con 0x7f40e4070b50 2024-01-21T03:20:04.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f40dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f40dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:04.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f40dbfff640 10 monclient: dump: 2024-01-21T03:20:04.109 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.109 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.109 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.109 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.109 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.109 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.109 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f40dbfff640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f40dbfff640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f40dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.066+0000 7f40dbfff640 1 -- 172.21.15.7:0/2426861665 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d4031700 con 0x7f40e4070b50 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40dbfff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40ea627640 5 monclient: authenticate success, global_id 4278 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40ea627640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40ea627640 4 set_mon_vals no callback set 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40dbfff640 1 -- 172.21.15.7:0/2426861665 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d402f050 con 0x7f40e4070b50 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40dbfff640 10 monclient: dump: 2024-01-21T03:20:04.110 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40ea627640 10 monclient: shutdown 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40ea627640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40ea627640 1 -- 172.21.15.7:0/2426861665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4070b50 msgr2=0x7f40e41401e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40ea627640 1 --2- 172.21.15.7:0/2426861665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4070b50 0x7f40e41401e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f40d40097a0 tx=0x7f40d402b7a0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40ea627640 1 -- 172.21.15.7:0/2426861665 shutdown_connections 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40ea627640 1 --2- 172.21.15.7:0/2426861665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f40e413d5a0 0x7f40e4140720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40ea627640 1 --2- 172.21.15.7:0/2426861665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4070b50 0x7f40e41401e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.067+0000 7f40ea627640 1 -- 172.21.15.7:0/2426861665 >> 172.21.15.7:0/2426861665 conn(0x7f40e40b0040 msgr2=0x7f40e40b0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7f40ea627640 1 -- 172.21.15.7:0/2426861665 shutdown_connections 2024-01-21T03:20:04.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7f40ea627640 1 -- 172.21.15.7:0/2426861665 wait complete. 2024-01-21T03:20:04.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3bdffb640 1 -- 172.21.15.7:0/2300227020 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b00317c0 con 0x7fc3b8038de0 2024-01-21T03:20:04.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3bdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3bdffb640 10 monclient: dump: 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3bdffb640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3bdffb640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3bdffb640 1 -- 172.21.15.7:0/2300227020 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b0031920 con 0x7fc3b8038de0 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3bdffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3bdffb640 1 -- 172.21.15.7:0/2300227020 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0031cb0 con 0x7fc3b8038de0 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3c5475640 5 monclient: authenticate success, global_id 4275 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3c5475640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3c5475640 4 set_mon_vals no callback set 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7fc3bdffb640 10 monclient: dump: 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.115 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.115 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7f40ea627640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.068+0000 7f40ea627640 10 monclient: monmap: 2024-01-21T03:20:04.115 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.115 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.115 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.115 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.115 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.115 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.116 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f40ea627640 1 librados: starting msgr at 2024-01-21T03:20:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f40ea627640 1 librados: starting objecter 2024-01-21T03:20:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7fc3c5475640 10 monclient: shutdown 2024-01-21T03:20:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7fc3c5475640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7fc3c5475640 1 -- 172.21.15.7:0/2300227020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b8038de0 msgr2=0x7fc3b80c5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7fc3c5475640 1 --2- 172.21.15.7:0/2300227020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b8038de0 0x7fc3b80c5290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00097a0 tx=0x7fc3b002b860 comp rx=0 tx=0).stop 2024-01-21T03:20:04.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f40ea627640 1 Processor -- start 2024-01-21T03:20:04.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7fc3c5475640 1 -- 172.21.15.7:0/2300227020 shutdown_connections 2024-01-21T03:20:04.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7fc3c5475640 1 --2- 172.21.15.7:0/2300227020 >> v2:172.21.15.177:3300/0 conn(0x7fc3b80391c0 0x7fc3b80c8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7fc3c5475640 1 --2- 172.21.15.7:0/2300227020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b8038de0 0x7fc3b80c5290 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7fc3c5475640 1 -- 172.21.15.7:0/2300227020 >> 172.21.15.7:0/2300227020 conn(0x7fc3b8006640 msgr2=0x7fc3b8006a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f40ea627640 1 -- start start 2024-01-21T03:20:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f40ea627640 1 librados: setting wanted keys 2024-01-21T03:20:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f40ea627640 1 librados: calling monclient init 2024-01-21T03:20:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f40ea627640 10 monclient: init 2024-01-21T03:20:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7fc3c5475640 1 -- 172.21.15.7:0/2300227020 shutdown_connections 2024-01-21T03:20:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7fc3c5475640 1 -- 172.21.15.7:0/2300227020 wait complete. 2024-01-21T03:20:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f1897fff640 1 --2- 172.21.15.7:0/488701857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1890035670 0x7f1890033c80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f189804ee90 tx=0x7f189806ca20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f1894ff9640 10 client.?.objecter ms_handle_connect 0x7f1890035670 2024-01-21T03:20:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f1894ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f40ea627640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f40ea627640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:20:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f40ea627640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4070b50 0x7f40e413c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.069+0000 7f40ea627640 10 monclient(hunting): picked mon.noname-a con 0x7f40e4070b50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7f40ea627640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40e4071dd0 0x7f40e41373e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7f40ea627640 10 monclient(hunting): picked mon.noname-b con 0x7f40e4071dd0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7f40ea627640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f40e413d5a0 0x7f40e40b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7f40ea627640 10 monclient(hunting): picked mon.noname-c con 0x7f40e413d5a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7f40ea627640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7f40ea627640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40b3270 con 0x7f40e4070b50 2024-01-21T03:20:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7f40ea627640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7f40ea627640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40b2f20 con 0x7f40e4071dd0 2024-01-21T03:20:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7f40ea627640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7f40ea627640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e413b090 con 0x7f40e413d5a0 2024-01-21T03:20:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7f40ea627640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7f40ea627640 10 monclient(hunting): authenticate will time out at 1350.385898s 2024-01-21T03:20:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7fc3c5475640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7fc3c5475640 10 monclient: monmap: 2024-01-21T03:20:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.122 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7fc3c5475640 1 librados: starting msgr at 2024-01-21T03:20:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.070+0000 7fc3c5475640 1 librados: starting objecter 2024-01-21T03:20:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 1 Processor -- start 2024-01-21T03:20:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 1 -- start start 2024-01-21T03:20:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 1 librados: setting wanted keys 2024-01-21T03:20:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 1 librados: calling monclient init 2024-01-21T03:20:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 10 monclient: init 2024-01-21T03:20:04.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7f502e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5030070a10 0x7f50301d8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:20:04.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3b8038de0 0x7fc3b802dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 10 monclient(hunting): picked mon.noname-b con 0x7fc3b8038de0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3b80c57d0 0x7fc3b80c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 10 monclient(hunting): picked mon.noname-c con 0x7fc3b80c57d0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b802fc30 0x7fc3b80300e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 10 monclient(hunting): picked mon.noname-a con 0x7fc3b802fc30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b802f7c0 con 0x7fc3b802fc30 2024-01-21T03:20:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b802f130 con 0x7fc3b8038de0 2024-01-21T03:20:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b8030620 con 0x7fc3b80c57d0 2024-01-21T03:20:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.071+0000 7fc3c5475640 10 monclient(hunting): authenticate will time out at 1350.387695s 2024-01-21T03:20:04.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.072+0000 7f502e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5030070a10 0x7f50301d8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44168/0 (socket says 172.21.15.7:44168) 2024-01-21T03:20:04.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.072+0000 7f502e59c640 1 -- 172.21.15.7:0/1482777839 learned_addr learned my addr 172.21.15.7:0/1482777839 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.072+0000 7f502e59c640 10 monclient(hunting): get_auth_request con 0x7f5030070a10 auth_method 0 2024-01-21T03:20:04.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.072+0000 7f502e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.072+0000 7f502e59c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.072+0000 7f502e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.073+0000 7f502e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.073+0000 7f502e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.073+0000 7f502e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f1894ff9640 1 -- 172.21.15.7:0/488701857 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1898067b80 con 0x7f1890035670 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f1894ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f1894ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f1894ff9640 10 monclient: dump: 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f1894ff9640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f1894ff9640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f1894ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f1894ff9640 1 -- 172.21.15.7:0/488701857 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189808e070 con 0x7f1890035670 2024-01-21T03:20:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f1894ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f1894ff9640 1 -- 172.21.15.7:0/488701857 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189806c370 con 0x7f1890035670 2024-01-21T03:20:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f189d62e640 5 monclient: authenticate success, global_id 4312 2024-01-21T03:20:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f189d62e640 10 monclient: _renew_subs 2024-01-21T03:20:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f189d62e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f189d62e640 1 -- 172.21.15.7:0/488701857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18900341c0 con 0x7f1890035670 2024-01-21T03:20:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f18967fc640 4 set_mon_vals no callback set 2024-01-21T03:20:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f1894ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f1894ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f1894ff9640 10 monclient: dump: 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f189d62e640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f189d62e640 10 monclient: _renew_subs 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f189d62e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f189d62e640 1 -- 172.21.15.7:0/488701857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18900bb4a0 con 0x7f1890035670 2024-01-21T03:20:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.074+0000 7f189d62e640 1 librados: init done 2024-01-21T03:20:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.075+0000 7f189d62e640 10 client.4312.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.075+0000 7f189d62e640 15 client.4312.objecter _session_command_op_assign -1 1 2024-01-21T03:20:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.075+0000 7f189d62e640 20 client.4312.objecter _get_session osd=-1 returning homeless 2024-01-21T03:20:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.075+0000 7f189d62e640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.075+0000 7f189d62e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.075+0000 7f189d62e640 1 -- 172.21.15.7:0/488701857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f18900393f0 con 0x7f1890035670 2024-01-21T03:20:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.077+0000 7f502e59c640 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2024-01-21T03:20:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.077+0000 7f502e59c640 1 -- 172.21.15.7:0/1482777839 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5030070df0 msgr2=0x7f5030136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.077+0000 7f502e59c640 1 --2- 172.21.15.7:0/1482777839 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5030070df0 0x7f5030136a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.077+0000 7f502e59c640 1 -- 172.21.15.7:0/1482777839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5030143400 msgr2=0x7f5030136fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.077+0000 7f502e59c640 1 --2- 172.21.15.7:0/1482777839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5030143400 0x7f5030136fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.077+0000 7f502e59c640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.077+0000 7f502e59c640 1 monclient: found mon.noname-a 2024-01-21T03:20:04.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.077+0000 7f502e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.077+0000 7f502e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.077+0000 7f502e59c640 1 -- 172.21.15.7:0/1482777839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5028009b30 con 0x7f5030070a10 2024-01-21T03:20:04.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.078+0000 7f6aa083e640 1 --2- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a94038db0 0x7f6a940c9cb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c075e20 tx=0x7f6a9c067a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.078+0000 7f6a997fa640 10 client.?.objecter ms_handle_connect 0x7f6a94038db0 2024-01-21T03:20:04.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.078+0000 7f6a997fa640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8257fff640 1 -- 172.21.15.7:0/2865264600 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826403ad10 con 0x7f8270071910 2024-01-21T03:20:04.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8257fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8257fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:04.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8257fff640 10 monclient: dump: 2024-01-21T03:20:04.132 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.132 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.132 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.132 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.132 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8257fff640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8257fff640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8257fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8257fff640 1 -- 172.21.15.7:0/2865264600 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826403ae70 con 0x7f8270071910 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8257fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8257fff640 1 -- 172.21.15.7:0/2865264600 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264044610 con 0x7f8270071910 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8257fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8257fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8257fff640 10 monclient: dump: 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.133 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f826d7fa640 4 set_mon_vals no callback set 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8275c46640 5 monclient: authenticate success, global_id 4274 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8275c46640 10 monclient: _renew_subs 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8275c46640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8275c46640 1 -- 172.21.15.7:0/2865264600 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827006cc10 con 0x7f8270071910 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8275c46640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8275c46640 10 monclient: _renew_subs 2024-01-21T03:20:04.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8275c46640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00effff640 1 -- 172.21.15.7:0/1842429924 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f4031740 con 0x7f00f80aed70 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00effff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00effff640 10 monclient: dump: 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.135 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8275c46640 1 -- 172.21.15.7:0/2865264600 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82700aab70 con 0x7f8270071910 2024-01-21T03:20:04.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f8275c46640 1 librados: init done 2024-01-21T03:20:04.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00effff640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00effff640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00effff640 1 -- 172.21.15.7:0/1842429924 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f40318a0 con 0x7f00f80aed70 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00effff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00effff640 1 -- 172.21.15.7:0/1842429924 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f4031c30 con 0x7f00f80aed70 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00effff640 10 monclient: dump: 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.138 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.138 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.138 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00ff7e2640 5 monclient: authenticate success, global_id 4271 2024-01-21T03:20:04.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00ff7e2640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.080+0000 7f00ff7e2640 4 set_mon_vals no callback set 2024-01-21T03:20:04.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f00ff7e2640 10 monclient: shutdown 2024-01-21T03:20:04.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f00ff7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f00ff7e2640 1 -- 172.21.15.7:0/1842429924 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00f80aed70 msgr2=0x7f00f80af150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f00ff7e2640 1 --2- 172.21.15.7:0/1842429924 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00f80aed70 0x7f00f80af150 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f00f4009870 tx=0x7f00f402b7e0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f00ff7e2640 1 -- 172.21.15.7:0/1842429924 shutdown_connections 2024-01-21T03:20:04.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f8275c46640 10 client.4274.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f8275c46640 15 client.4274.objecter _session_command_op_assign -1 1 2024-01-21T03:20:04.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f8275c46640 20 client.4274.objecter _get_session osd=-1 returning homeless 2024-01-21T03:20:04.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f8275c46640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f8275c46640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f8275c46640 1 -- 172.21.15.7:0/2865264600 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f827004e970 con 0x7f8270071910 2024-01-21T03:20:04.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f00ff7e2640 1 --2- 172.21.15.7:0/1842429924 >> v2:172.21.15.177:3300/0 conn(0x7f00f80afff0 0x7f00f8142640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f00ff7e2640 1 --2- 172.21.15.7:0/1842429924 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00f80aed70 0x7f00f80af150 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.081+0000 7f00ff7e2640 1 -- 172.21.15.7:0/1842429924 >> 172.21.15.7:0/1842429924 conn(0x7f00f80b3a80 msgr2=0x7f00f80b3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.082+0000 7f502e59c640 1 --2- 172.21.15.7:0/1482777839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5030070a10 0x7f50301d8900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f502000d800 tx=0x7f502000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.082+0000 7f5026ffd640 10 client.?.objecter ms_handle_connect 0x7f5030070a10 2024-01-21T03:20:04.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.082+0000 7f5026ffd640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.083+0000 7f00ff7e2640 1 -- 172.21.15.7:0/1842429924 shutdown_connections 2024-01-21T03:20:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.083+0000 7f00ff7e2640 1 -- 172.21.15.7:0/1842429924 wait complete. 2024-01-21T03:20:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.084+0000 7f00ff7e2640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.084+0000 7f00ff7e2640 10 monclient: monmap: 2024-01-21T03:20:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.142 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.142 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.142 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.142 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.084+0000 7f00ff7e2640 1 librados: starting msgr at 2024-01-21T03:20:04.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.084+0000 7f00ff7e2640 1 librados: starting objecter 2024-01-21T03:20:04.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.084+0000 7f00ff7e2640 1 Processor -- start 2024-01-21T03:20:04.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.084+0000 7f40e9625640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4070b50 0x7f40e413c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.085+0000 7f40e9625640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4070b50 0x7f40e413c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44182/0 (socket says 172.21.15.7:44182) 2024-01-21T03:20:04.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.085+0000 7f40e9625640 1 -- 172.21.15.7:0/826364513 learned_addr learned my addr 172.21.15.7:0/826364513 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.085+0000 7f40e9625640 10 monclient(hunting): get_auth_request con 0x7f40e4070b50 auth_method 0 2024-01-21T03:20:04.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.085+0000 7f40e9625640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.085+0000 7f40e9625640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.085+0000 7f40e9625640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.084+0000 7f00ff7e2640 1 -- start start 2024-01-21T03:20:04.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.084+0000 7f00ff7e2640 1 librados: setting wanted keys 2024-01-21T03:20:04.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.084+0000 7f00ff7e2640 1 librados: calling monclient init 2024-01-21T03:20:04.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.084+0000 7f00ff7e2640 10 monclient: init 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.085+0000 7f00ff7e2640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.085+0000 7f00ff7e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.085+0000 7f00ff7e2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00f80aed70 0x7f00f8144580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.085+0000 7f00ff7e2640 10 monclient(hunting): picked mon.noname-c con 0x7f00f80aed70 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.085+0000 7f00ff7e2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00f80afff0 0x7f00f8144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.085+0000 7f00ff7e2640 10 monclient(hunting): picked mon.noname-b con 0x7f00f80afff0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.085+0000 7f00ff7e2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00f8142b80 0x7f00f81d73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.086+0000 7f00ff7e2640 10 monclient(hunting): picked mon.noname-a con 0x7f00f8142b80 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.086+0000 7f00ff7e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.086+0000 7f00ff7e2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8145bd0 con 0x7f00f8142b80 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.086+0000 7f00ff7e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.086+0000 7f00ff7e2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8145eb0 con 0x7f00f80afff0 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.086+0000 7f00ff7e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.086+0000 7f00ff7e2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f81437e0 con 0x7f00f80aed70 2024-01-21T03:20:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.086+0000 7f00ff7e2640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.086+0000 7f00ff7e2640 10 monclient(hunting): authenticate will time out at 1350.401873s 2024-01-21T03:20:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.086+0000 7f40e9625640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.086+0000 7f40e9625640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.086+0000 7f40e9625640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.092+0000 7f40e9625640 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2024-01-21T03:20:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.092+0000 7f40e9625640 1 -- 172.21.15.7:0/826364513 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f40e413d5a0 msgr2=0x7f40e40b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.092+0000 7f40e9625640 1 --2- 172.21.15.7:0/826364513 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f40e413d5a0 0x7f40e40b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.092+0000 7f40e9625640 1 -- 172.21.15.7:0/826364513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40e4071dd0 msgr2=0x7f40e41373e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.092+0000 7f40e9625640 1 --2- 172.21.15.7:0/826364513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40e4071dd0 0x7f40e41373e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.092+0000 7f40e9625640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.092+0000 7f40e9625640 1 monclient: found mon.noname-a 2024-01-21T03:20:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.092+0000 7f40e9625640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.092+0000 7f40e9625640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.092+0000 7f40e9625640 1 -- 172.21.15.7:0/826364513 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d4009400 con 0x7f40e4070b50 2024-01-21T03:20:04.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.094+0000 7f00fdd7f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00f8142b80 0x7f00f81d73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.095+0000 7f40d9ffb640 10 client.?.objecter ms_handle_connect 0x7f40e4070b50 2024-01-21T03:20:04.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.095+0000 7f40d9ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.095+0000 7f40e9625640 1 --2- 172.21.15.7:0/826364513 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4070b50 0x7f40e413c580 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f40d402b7e0 tx=0x7f40d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.096+0000 7fc3bf7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b802fc30 0x7fc3b80300e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.096+0000 7f00fdd7f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00f8142b80 0x7f00f81d73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44198/0 (socket says 172.21.15.7:44198) 2024-01-21T03:20:04.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.096+0000 7f00fdd7f640 1 -- 172.21.15.7:0/3891581075 learned_addr learned my addr 172.21.15.7:0/3891581075 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.096+0000 7f00fdd7f640 10 monclient(hunting): get_auth_request con 0x7f00f8142b80 auth_method 0 2024-01-21T03:20:04.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.096+0000 7f00fdd7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.096+0000 7f00fdd7f640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.096+0000 7f00fdd7f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.099+0000 7f1894ff9640 1 -- 172.21.15.7:0/488701857 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f189806a630 con 0x7f1890035670 2024-01-21T03:20:04.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.099+0000 7f1894ff9640 1 --2- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f187c06c860 0x7f187c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.100+0000 7f18977fe640 1 --2- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f187c06c860 0x7f187c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.100+0000 7f18977fe640 10 monclient: get_auth_request con 0x7f187c06c860 auth_method 0 2024-01-21T03:20:04.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.100+0000 7f1894ff9640 10 client.4312.objecter ms_handle_connect 0x7f187c06c860 2024-01-21T03:20:04.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.100+0000 7f18977fe640 1 --2- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f187c06c860 0x7f187c06ed20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f18900cc1e0 tx=0x7f18840079b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.100+0000 7fc3bf7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b802fc30 0x7fc3b80300e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44196/0 (socket says 172.21.15.7:44196) 2024-01-21T03:20:04.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.100+0000 7fc3bf7fe640 1 -- 172.21.15.7:0/648551170 learned_addr learned my addr 172.21.15.7:0/648551170 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.100+0000 7fc3bf7fe640 10 monclient(hunting): get_auth_request con 0x7fc3b802fc30 auth_method 0 2024-01-21T03:20:04.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.100+0000 7fc3bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.100+0000 7fc3bf7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.100+0000 7fc3bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.101+0000 7f5026ffd640 1 -- 172.21.15.7:0/1482777839 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502000b6c0 con 0x7f5030070a10 2024-01-21T03:20:04.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.101+0000 7f5026ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f5026ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:04.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f5026ffd640 10 monclient: dump: 2024-01-21T03:20:04.152 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.152 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.152 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.152 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.152 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f5026ffd640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f5026ffd640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f5026ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f5026ffd640 1 -- 172.21.15.7:0/1482777839 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502000b820 con 0x7f5030070a10 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f5026ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f5026ffd640 1 -- 172.21.15.7:0/1482777839 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502000bbb0 con 0x7f5030070a10 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 5 monclient: authenticate success, global_id 4281 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 10 monclient: _renew_subs 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 1 -- 172.21.15.7:0/1482777839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5030137500 con 0x7f5030070a10 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f5026ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f502d59a640 4 set_mon_vals no callback set 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f5026ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f5026ffd640 10 monclient: dump: 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 10 monclient: _renew_subs 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 1 -- 172.21.15.7:0/1482777839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50301d9340 con 0x7f5030070a10 2024-01-21T03:20:04.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 1 librados: init done 2024-01-21T03:20:04.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 10 client.4281.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 15 client.4281.objecter _session_command_op_assign -1 1 2024-01-21T03:20:04.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 20 client.4281.objecter _get_session osd=-1 returning homeless 2024-01-21T03:20:04.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.102+0000 7f50349e2640 1 -- 172.21.15.7:0/1482777839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f50300b02c0 con 0x7f5030070a10 2024-01-21T03:20:04.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.103+0000 7f00fdd7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.103+0000 7f00fdd7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.103+0000 7f00fdd7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.103+0000 7fc3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.103+0000 7fc3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.103+0000 7fc3bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 1 -- 172.21.15.7:0/488701857 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f18980d2910 con 0x7f1890035670 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 10 client.4312.objecter ms_dispatch 0x7f1898000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 3 client.4312.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 3 client.4312.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 10 client.4312.objecter checking command 1 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 1 --2- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] conn(0x7f187c072c40 0x7f187c075060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 20 client.4312.objecter _get_session s=0x7f187c072570 osd=3 3 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 20 client.4312.objecter put_session s=0x7f187c072570 osd=3 3 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 15 client.4312.objecter _session_command_op_remove -1 1 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 20 client.4312.objecter _get_session s=0x7f187c072570 osd=3 3 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 20 client.4312.objecter get_session s=0x7f187c072570 osd=3 3 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 15 client.4312.objecter _session_command_op_assign 3 1 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 20 client.4312.objecter put_session s=0x7f187c072570 osd=3 4 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 10 client.4312.objecter _send_command 1 2024-01-21T03:20:04.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 1 -- 172.21.15.7:0/488701857 --> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f187c075710 con 0x7f187c072c40 2024-01-21T03:20:04.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.104+0000 7f1894ff9640 20 client.4312.objecter dump_active .. 0 homeless 2024-01-21T03:20:04.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.105+0000 7f189ce2d640 1 --2- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] conn(0x7f187c072c40 0x7f187c075060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.105+0000 7fc3beffd640 1 --2- 172.21.15.7:0/648551170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3b8038de0 0x7fc3b802dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.105+0000 7fc3beffd640 10 monclient(hunting): get_auth_request con 0x7fc3b8038de0 auth_method 0 2024-01-21T03:20:04.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.105+0000 7fc3beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.105+0000 7fc3beffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.105+0000 7fc3beffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.105+0000 7f189ce2d640 10 monclient: get_auth_request con 0x7f187c072c40 auth_method 0 2024-01-21T03:20:04.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7fc3bf7fe640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2024-01-21T03:20:04.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7fc3bf7fe640 1 -- 172.21.15.7:0/648551170 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3b80c57d0 msgr2=0x7fc3b80c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7fc3bf7fe640 1 --2- 172.21.15.7:0/648551170 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3b80c57d0 0x7fc3b80c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7fc3bf7fe640 1 -- 172.21.15.7:0/648551170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3b8038de0 msgr2=0x7fc3b802dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7fc3bf7fe640 1 --2- 172.21.15.7:0/648551170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3b8038de0 0x7fc3b802dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7fc3bf7fe640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7fc3bf7fe640 1 monclient: found mon.noname-a 2024-01-21T03:20:04.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7fc3bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7fc3bf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7fc3bf7fe640 1 -- 172.21.15.7:0/648551170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b0009400 con 0x7fc3b802fc30 2024-01-21T03:20:04.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0030e20 con 0x7f3ed40047c0 2024-01-21T03:20:04.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7f3ee5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7f3ee5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:04.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7f3ee5d9b640 10 monclient: dump: 2024-01-21T03:20:04.161 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.161 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.161 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.161 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.161 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed00058d0 con 0x7f3ed40047c0 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7f3eece00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7f3ee5d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7f3eece00640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7f3eece00640 4 set_mon_vals no callback set 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7f3ee5d9b640 1 -- 172.21.15.7:0/3326937985 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0005c80 con 0x7f3ed40047c0 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7f3ee5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7f3ee5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.106+0000 7f3ee5d9b640 10 monclient: dump: 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.107+0000 7f3eece00640 10 monclient: shutdown 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.107+0000 7f3eece00640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.107+0000 7f3eece00640 1 -- 172.21.15.7:0/3326937985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed40047c0 msgr2=0x7f3ed4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.107+0000 7f3eece00640 1 --2- 172.21.15.7:0/3326937985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed40047c0 0x7f3ed4004c40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0007c00 tx=0x7f3ed40057f0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.107+0000 7f3ee6d9d640 1 -- 172.21.15.7:0/3326937985 reap_dead start 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.107+0000 7f3eece00640 1 -- 172.21.15.7:0/3326937985 shutdown_connections 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.107+0000 7f3eece00640 1 -- 172.21.15.7:0/3326937985 >> 172.21.15.7:0/3326937985 conn(0x7f3ee806aa50 msgr2=0x7f3ee806ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.107+0000 7f3eece00640 1 -- 172.21.15.7:0/3326937985 shutdown_connections 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.107+0000 7f3eece00640 1 -- 172.21.15.7:0/3326937985 wait complete. 2024-01-21T03:20:04.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.108+0000 7f3eece00640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.108+0000 7f3eece00640 10 monclient: monmap: 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.109+0000 7f3eece00640 1 librados: starting msgr at 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.109+0000 7f3eece00640 1 librados: starting objecter 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.109+0000 7f3eece00640 1 Processor -- start 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.109+0000 7f3eece00640 1 -- start start 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.109+0000 7f3eece00640 1 librados: setting wanted keys 2024-01-21T03:20:04.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.109+0000 7f3eece00640 1 librados: calling monclient init 2024-01-21T03:20:04.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.109+0000 7f3eece00640 10 monclient: init 2024-01-21T03:20:04.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f1894ff9640 10 client.4312.objecter ms_handle_connect 0x7f187c072c40 2024-01-21T03:20:04.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f189ce2d640 1 --2- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] conn(0x7f187c072c40 0x7f187c075060 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ee80b2d60 0x7f3ee80b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 10 monclient(hunting): picked mon.noname-b con 0x7f3ee80b2d60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3ee8130650 0x7f3ee8130ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 10 monclient(hunting): picked mon.noname-c con 0x7f3ee8130650 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ee81d35b0 0x7f3ee81d3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 10 monclient(hunting): picked mon.noname-a con 0x7f3ee81d35b0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80b3140 con 0x7f3ee81d35b0 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee813fb30 con 0x7f3ee80b2d60 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80b26e0 con 0x7f3ee8130650 2024-01-21T03:20:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.110+0000 7f3eece00640 10 monclient(hunting): authenticate will time out at 1350.426146s 2024-01-21T03:20:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.111+0000 7f00fdd7f640 10 monclient(hunting): handle_auth_done global_id 4287 payload 824 2024-01-21T03:20:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.111+0000 7f00fdd7f640 1 -- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00f80aed70 msgr2=0x7f00f8144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.111+0000 7f00fdd7f640 1 --2- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f00f80aed70 0x7f00f8144580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.111+0000 7f00fdd7f640 1 -- 172.21.15.7:0/3891581075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00f80afff0 msgr2=0x7f00f8144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.111+0000 7f00fdd7f640 1 --2- 172.21.15.7:0/3891581075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00f80afff0 0x7f00f8144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.111+0000 7f00fdd7f640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.111+0000 7f00fdd7f640 1 monclient: found mon.noname-a 2024-01-21T03:20:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.111+0000 7f00fdd7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.111+0000 7f00fdd7f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.111+0000 7f00fdd7f640 1 -- 172.21.15.7:0/3891581075 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f40094d0 con 0x7f00f8142b80 2024-01-21T03:20:04.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.114+0000 7f3ee659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ee80b2d60 0x7f3ee80b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.115+0000 7f3ee659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ee80b2d60 0x7f3ee80b1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36646/0 (socket says 172.21.15.7:36646) 2024-01-21T03:20:04.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.115+0000 7f3ee659c640 1 -- 172.21.15.7:0/1828190540 learned_addr learned my addr 172.21.15.7:0/1828190540 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.115+0000 7f3ee659c640 10 monclient(hunting): get_auth_request con 0x7f3ee80b2d60 auth_method 0 2024-01-21T03:20:04.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.115+0000 7f3ee659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.115+0000 7f3ee659c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.115+0000 7f3ee659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.115+0000 7f3ee659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.115+0000 7f3ee659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.115+0000 7f3ee659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.116+0000 7f8257fff640 1 -- 172.21.15.7:0/2865264600 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8264044770 con 0x7f8270071910 2024-01-21T03:20:04.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.117+0000 7f8257fff640 1 --2- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f825c06c8e0 0x7f825c06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.117+0000 7f826f7fe640 1 --2- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f825c06c8e0 0x7f825c06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.117+0000 7f826f7fe640 10 monclient: get_auth_request con 0x7f825c06c8e0 auth_method 0 2024-01-21T03:20:04.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.118+0000 7f8257fff640 10 client.4274.objecter ms_handle_connect 0x7f825c06c8e0 2024-01-21T03:20:04.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.118+0000 7f826f7fe640 1 --2- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f825c06c8e0 0x7f825c06eda0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8260001560 tx=0x7f8260007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.118+0000 7f3ee659c640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2024-01-21T03:20:04.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.118+0000 7f3ee659c640 1 -- 172.21.15.7:0/1828190540 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3ee8130650 msgr2=0x7f3ee8130ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.118+0000 7f3ee659c640 1 --2- 172.21.15.7:0/1828190540 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3ee8130650 0x7f3ee8130ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.118+0000 7f3ee659c640 1 -- 172.21.15.7:0/1828190540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ee81d35b0 msgr2=0x7f3ee81d3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.118+0000 7f3ee659c640 1 --2- 172.21.15.7:0/1828190540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ee81d35b0 0x7f3ee81d3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.118+0000 7f3ee659c640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.118+0000 7f3ee659c640 1 monclient: found mon.noname-b 2024-01-21T03:20:04.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.118+0000 7f3ee659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.118+0000 7f3ee659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.118+0000 7f3ee659c640 1 -- 172.21.15.7:0/1828190540 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee813fd60 con 0x7f3ee80b2d60 2024-01-21T03:20:04.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 1 -- 172.21.15.7:0/2865264600 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f826402d040 con 0x7f8270071910 2024-01-21T03:20:04.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 10 client.4274.objecter ms_dispatch 0x7f8270000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:04.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 3 client.4274.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:04.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 3 client.4274.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:04.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 10 client.4274.objecter checking command 1 2024-01-21T03:20:04.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 1 --2- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] conn(0x7f825c072cc0 0x7f825c0750e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 20 client.4274.objecter _get_session s=0x7f825c0725f0 osd=0 3 2024-01-21T03:20:04.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 20 client.4274.objecter put_session s=0x7f825c0725f0 osd=0 3 2024-01-21T03:20:04.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 15 client.4274.objecter _session_command_op_remove -1 1 2024-01-21T03:20:04.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 20 client.4274.objecter _get_session s=0x7f825c0725f0 osd=0 3 2024-01-21T03:20:04.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 20 client.4274.objecter get_session s=0x7f825c0725f0 osd=0 3 2024-01-21T03:20:04.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 15 client.4274.objecter _session_command_op_assign 0 1 2024-01-21T03:20:04.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 20 client.4274.objecter put_session s=0x7f825c0725f0 osd=0 4 2024-01-21T03:20:04.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 10 client.4274.objecter _send_command 1 2024-01-21T03:20:04.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 1 -- 172.21.15.7:0/2865264600 --> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f825c075790 con 0x7f825c072cc0 2024-01-21T03:20:04.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f8257fff640 20 client.4274.objecter dump_active .. 0 homeless 2024-01-21T03:20:04.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f3ee659c640 1 --2- 172.21.15.7:0/1828190540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ee80b2d60 0x7f3ee80b1fc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800c850 tx=0x7f3ed800cd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f3edeffd640 10 client.?.objecter ms_handle_connect 0x7f3ee80b2d60 2024-01-21T03:20:04.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.120+0000 7f3edeffd640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.121+0000 7fc3bf7fe640 1 --2- 172.21.15.7:0/648551170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b802fc30 0x7fc3b80300e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0050150 tx=0x7fc3c0067a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.121+0000 7fc3a77fe640 10 client.?.objecter ms_handle_connect 0x7fc3b802fc30 2024-01-21T03:20:04.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.121+0000 7fc3a77fe640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.121+0000 7f826ffff640 1 --2- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] conn(0x7f825c072cc0 0x7f825c0750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.121+0000 7f826ffff640 10 monclient: get_auth_request con 0x7f825c072cc0 auth_method 0 2024-01-21T03:20:04.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3edeffd640 1 -- 172.21.15.7:0/1828190540 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8004220 con 0x7f3ee80b2d60 2024-01-21T03:20:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3edeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3edeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3edeffd640 10 monclient: dump: 2024-01-21T03:20:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.176 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3edeffd640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3edeffd640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3edeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3edeffd640 1 -- 172.21.15.7:0/1828190540 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed8004380 con 0x7f3ee80b2d60 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3edeffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3edeffd640 1 -- 172.21.15.7:0/1828190540 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80079f0 con 0x7f3ee80b2d60 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3edeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3edeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3edeffd640 10 monclient: dump: 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3eece00640 5 monclient: authenticate success, global_id 4318 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3eece00640 10 monclient: _renew_subs 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3eece00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3eece00640 1 -- 172.21.15.7:0/1828190540 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee8140040 con 0x7f3ee80b2d60 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3ee559a640 4 set_mon_vals no callback set 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3eece00640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3eece00640 10 monclient: _renew_subs 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3eece00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3eece00640 1 -- 172.21.15.7:0/1828190540 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee81d3f70 con 0x7f3ee80b2d60 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.124+0000 7f3eece00640 1 librados: init done 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.125+0000 7f3eece00640 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.125+0000 7f3eece00640 15 client.4318.objecter _session_command_op_assign -1 1 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.125+0000 7f3eece00640 20 client.4318.objecter _get_session osd=-1 returning homeless 2024-01-21T03:20:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.125+0000 7f3eece00640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.125+0000 7f3eece00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.125+0000 7f3eece00640 1 -- 172.21.15.7:0/1828190540 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3ee813ba00 con 0x7f3ee80b2d60 2024-01-21T03:20:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.127+0000 7f826ffff640 1 --2- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] conn(0x7f825c072cc0 0x7f825c0750e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.127+0000 7f8257fff640 10 client.4274.objecter ms_handle_connect 0x7f825c072cc0 2024-01-21T03:20:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.127+0000 7f6a997fa640 1 -- 172.21.15.7:0/2784924632 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c093470 con 0x7f6a94038db0 2024-01-21T03:20:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.127+0000 7f6a997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.127+0000 7f6a997fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.127+0000 7f6a997fa640 10 monclient: dump: 2024-01-21T03:20:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.127+0000 7f6a997fa640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.127+0000 7f6a997fa640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.127+0000 7f6a997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6aa1840640 5 monclient: authenticate success, global_id 4277 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6aa1840640 10 monclient: _renew_subs 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6aa1840640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6aa1840640 1 -- 172.21.15.7:0/2784924632 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a94032f80 con 0x7f6a94038db0 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6a997fa640 1 -- 172.21.15.7:0/2784924632 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9c0935d0 con 0x7f6a94038db0 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6a997fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6a997fa640 1 -- 172.21.15.7:0/2784924632 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c093980 con 0x7f6a94038db0 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6a997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6a997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6a997fa640 10 monclient: dump: 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6a9affd640 4 set_mon_vals no callback set 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6aa1840640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6aa1840640 10 monclient: _renew_subs 2024-01-21T03:20:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6aa1840640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6aa1840640 1 -- 172.21.15.7:0/2784924632 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a940ca1f0 con 0x7f6a94038db0 2024-01-21T03:20:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.128+0000 7f6aa1840640 1 librados: init done 2024-01-21T03:20:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.129+0000 7f6aa1840640 10 client.4277.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.129+0000 7f6aa1840640 15 client.4277.objecter _session_command_op_assign -1 1 2024-01-21T03:20:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.129+0000 7f6aa1840640 20 client.4277.objecter _get_session osd=-1 returning homeless 2024-01-21T03:20:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.129+0000 7f6aa1840640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.129+0000 7f6aa1840640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.129+0000 7f6aa1840640 1 -- 172.21.15.7:0/2784924632 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6a94036980 con 0x7f6a94038db0 2024-01-21T03:20:04.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.143+0000 7f3edeffd640 1 -- 172.21.15.7:0/1828190540 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f3ed8007b50 con 0x7f3ee80b2d60 2024-01-21T03:20:04.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.144+0000 7f3edeffd640 1 --2- 172.21.15.7:0/1828190540 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f3ec806c890 0x7f3ec806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.144+0000 7f3edffff640 1 --2- 172.21.15.7:0/1828190540 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f3ec806c890 0x7f3ec806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.144+0000 7f3edffff640 10 monclient: get_auth_request con 0x7f3ec806c890 auth_method 0 2024-01-21T03:20:04.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.144+0000 7f5026ffd640 1 -- 172.21.15.7:0/1482777839 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f502000bd10 con 0x7f5030070a10 2024-01-21T03:20:04.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.145+0000 7f3edffff640 1 --2- 172.21.15.7:0/1828190540 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f3ec806c890 0x7f3ec806ed50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0038000 tx=0x7f3ed003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.145+0000 7f3edeffd640 10 client.4318.objecter ms_handle_connect 0x7f3ec806c890 2024-01-21T03:20:04.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.145+0000 7f5026ffd640 1 --2- 172.21.15.7:0/1482777839 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f501006c8e0 0x7f501006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.145+0000 7f5027fff640 1 --2- 172.21.15.7:0/1482777839 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f501006c8e0 0x7f501006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.145+0000 7f5027fff640 10 monclient: get_auth_request con 0x7f501006c8e0 auth_method 0 2024-01-21T03:20:04.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.146+0000 7f5027fff640 1 --2- 172.21.15.7:0/1482777839 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f501006c8e0 0x7f501006eda0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5030142ea0 tx=0x7f5018006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.146+0000 7f5026ffd640 10 client.4281.objecter ms_handle_connect 0x7f501006c8e0 2024-01-21T03:20:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.146+0000 7f00fdd7f640 1 --2- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00f8142b80 0x7f00f81d73c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f00f000b990 tx=0x7f00f000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.146+0000 7f00edffb640 10 client.?.objecter ms_handle_connect 0x7f00f8142b80 2024-01-21T03:20:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.146+0000 7f00edffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 1 -- 172.21.15.7:0/1828190540 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3ed8058aa0 con 0x7f3ee80b2d60 2024-01-21T03:20:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 10 client.4318.objecter ms_dispatch 0x7f3ee8000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 3 client.4318.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 3 client.4318.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 10 client.4318.objecter checking command 1 2024-01-21T03:20:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 1 --2- 172.21.15.7:0/1828190540 >> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] conn(0x7f3ec8072c70 0x7f3ec8075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 20 client.4318.objecter _get_session s=0x7f3ec80725a0 osd=4 3 2024-01-21T03:20:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 20 client.4318.objecter put_session s=0x7f3ec80725a0 osd=4 3 2024-01-21T03:20:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 15 client.4318.objecter _session_command_op_remove -1 1 2024-01-21T03:20:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 20 client.4318.objecter _get_session s=0x7f3ec80725a0 osd=4 3 2024-01-21T03:20:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 20 client.4318.objecter get_session s=0x7f3ec80725a0 osd=4 3 2024-01-21T03:20:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 15 client.4318.objecter _session_command_op_assign 4 1 2024-01-21T03:20:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 20 client.4318.objecter put_session s=0x7f3ec80725a0 osd=4 4 2024-01-21T03:20:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 10 client.4318.objecter _send_command 1 2024-01-21T03:20:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 1 -- 172.21.15.7:0/1828190540 --> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ec8075740 con 0x7f3ec8072c70 2024-01-21T03:20:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f3edeffd640 20 client.4318.objecter dump_active .. 0 homeless 2024-01-21T03:20:04.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40d9ffb640 1 -- 172.21.15.7:0/826364513 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d4031ef0 con 0x7f40e4070b50 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40d9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40d9ffb640 10 monclient: dump: 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40d9ffb640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40d9ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40d9ffb640 1 -- 172.21.15.7:0/826364513 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d40044c0 con 0x7f40e4070b50 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40d9ffb640 1 -- 172.21.15.7:0/826364513 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d4003a60 con 0x7f40e4070b50 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40d9ffb640 10 monclient: dump: 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.191 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.150+0000 7f40db7fe640 4 set_mon_vals no callback set 2024-01-21T03:20:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.151+0000 7f40ea627640 5 monclient: authenticate success, global_id 4284 2024-01-21T03:20:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.151+0000 7f40ea627640 10 monclient: _renew_subs 2024-01-21T03:20:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.151+0000 7f40ea627640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.151+0000 7f40ea627640 1 -- 172.21.15.7:0/826364513 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e41321e0 con 0x7f40e4070b50 2024-01-21T03:20:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.151+0000 7f40ea627640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.151+0000 7f40ea627640 10 monclient: _renew_subs 2024-01-21T03:20:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.151+0000 7f40ea627640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.151+0000 7f40ea627640 1 -- 172.21.15.7:0/826364513 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e40b28b0 con 0x7f40e4070b50 2024-01-21T03:20:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.151+0000 7f40ea627640 1 librados: init done 2024-01-21T03:20:04.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.151+0000 7f3ee6d9d640 1 --2- 172.21.15.7:0/1828190540 >> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] conn(0x7f3ec8072c70 0x7f3ec8075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.151+0000 7f3ee6d9d640 10 monclient: get_auth_request con 0x7f3ec8072c70 auth_method 0 2024-01-21T03:20:04.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 1 -- 172.21.15.7:0/1482777839 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f50200597e0 con 0x7f5030070a10 2024-01-21T03:20:04.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 10 client.4281.objecter ms_dispatch 0x7f5030000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:04.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 3 client.4281.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:04.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 3 client.4281.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:04.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f40ea627640 10 client.4284.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f40ea627640 15 client.4284.objecter _session_command_op_assign -1 1 2024-01-21T03:20:04.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f40ea627640 20 client.4284.objecter _get_session osd=-1 returning homeless 2024-01-21T03:20:04.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f40ea627640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f40ea627640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f40ea627640 1 -- 172.21.15.7:0/826364513 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f40e4071180 con 0x7f40e4070b50 2024-01-21T03:20:04.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 10 client.4281.objecter checking command 1 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 1 --2- 172.21.15.7:0/1482777839 >> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] conn(0x7f5010072cc0 0x7f50100750e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 20 client.4281.objecter _get_session s=0x7f50100725f0 osd=6 3 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 20 client.4281.objecter put_session s=0x7f50100725f0 osd=6 3 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 15 client.4281.objecter _session_command_op_remove -1 1 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 20 client.4281.objecter _get_session s=0x7f50100725f0 osd=6 3 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 20 client.4281.objecter get_session s=0x7f50100725f0 osd=6 3 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 15 client.4281.objecter _session_command_op_assign 6 1 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 20 client.4281.objecter put_session s=0x7f50100725f0 osd=6 4 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 10 client.4281.objecter _send_command 1 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 1 -- 172.21.15.7:0/1482777839 --> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5010079ba0 con 0x7f5010072cc0 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 20 client.4281.objecter dump_active .. 0 homeless 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 1 -- 172.21.15.7:0/1482777839 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5020055420 con 0x7f5030070a10 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f5026ffd640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.152+0000 7f502ed9d640 1 --2- 172.21.15.7:0/1482777839 >> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] conn(0x7f5010072cc0 0x7f50100750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.153+0000 7f502ed9d640 10 monclient: get_auth_request con 0x7f5010072cc0 auth_method 0 2024-01-21T03:20:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.154+0000 7f6a997fa640 1 -- 172.21.15.7:0/2784924632 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6a9c090070 con 0x7f6a94038db0 2024-01-21T03:20:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.154+0000 7f6a997fa640 1 --2- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6a8006c860 0x7f6a8006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.154+0000 7f6a9bfff640 1 --2- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6a8006c860 0x7f6a8006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.154+0000 7f6a9bfff640 10 monclient: get_auth_request con 0x7f6a8006c860 auth_method 0 2024-01-21T03:20:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.155+0000 7f6a9bfff640 1 --2- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6a8006c860 0x7f6a8006ed20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6a88036000 tx=0x7f6a88030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.155+0000 7f6a997fa640 10 client.4277.objecter ms_handle_connect 0x7f6a8006c860 2024-01-21T03:20:04.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.155+0000 7f3edeffd640 10 client.4318.objecter ms_handle_connect 0x7f3ec8072c70 2024-01-21T03:20:04.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.155+0000 7f3ee6d9d640 1 --2- 172.21.15.7:0/1828190540 >> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] conn(0x7f3ec8072c70 0x7f3ec8075090 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.156+0000 7fc3a77fe640 1 -- 172.21.15.7:0/648551170 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0093470 con 0x7fc3b802fc30 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.156+0000 7fc3a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a77fe640 10 monclient: dump: 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a77fe640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a77fe640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a77fe640 1 -- 172.21.15.7:0/648551170 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c00935d0 con 0x7fc3b802fc30 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a77fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a77fe640 1 -- 172.21.15.7:0/648551170 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0093780 con 0x7fc3b802fc30 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a77fe640 10 monclient: dump: 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3c5475640 5 monclient: authenticate success, global_id 4290 2024-01-21T03:20:04.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3c5475640 10 monclient: _renew_subs 2024-01-21T03:20:04.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3c5475640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3c5475640 1 -- 172.21.15.7:0/648551170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b802f3a0 con 0x7fc3b802fc30 2024-01-21T03:20:04.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3bd7fa640 4 set_mon_vals no callback set 2024-01-21T03:20:04.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7f502ed9d640 1 --2- 172.21.15.7:0/1482777839 >> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] conn(0x7f5010072cc0 0x7f50100750e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7f5026ffd640 10 client.4281.objecter ms_handle_connect 0x7f5010072cc0 2024-01-21T03:20:04.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3c5475640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3c5475640 10 monclient: _renew_subs 2024-01-21T03:20:04.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3c5475640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3c5475640 1 -- 172.21.15.7:0/648551170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b802da10 con 0x7fc3b802fc30 2024-01-21T03:20:04.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3c5475640 1 librados: init done 2024-01-21T03:20:04.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a57fa640 10 client.4290.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a57fa640 15 client.4290.objecter _session_command_op_assign -1 1 2024-01-21T03:20:04.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a57fa640 20 client.4290.objecter _get_session osd=-1 returning homeless 2024-01-21T03:20:04.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a57fa640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a57fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.157+0000 7fc3a57fa640 1 -- 172.21.15.7:0/648551170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc398002910 con 0x7fc3b802fc30 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 1 -- 172.21.15.7:0/2784924632 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6a9c0d29f0 con 0x7f6a94038db0 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 10 client.4277.objecter ms_dispatch 0x7f6a9c000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 3 client.4277.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 3 client.4277.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 10 client.4277.objecter checking command 1 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 1 --2- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] conn(0x7f6a80072c40 0x7f6a80075060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 20 client.4277.objecter _get_session s=0x7f6a80072570 osd=1 3 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 20 client.4277.objecter put_session s=0x7f6a80072570 osd=1 3 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 15 client.4277.objecter _session_command_op_remove -1 1 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 20 client.4277.objecter _get_session s=0x7f6a80072570 osd=1 3 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 20 client.4277.objecter get_session s=0x7f6a80072570 osd=1 3 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 15 client.4277.objecter _session_command_op_assign 1 1 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 20 client.4277.objecter put_session s=0x7f6a80072570 osd=1 4 2024-01-21T03:20:04.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 10 client.4277.objecter _send_command 1 2024-01-21T03:20:04.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 1 -- 172.21.15.7:0/2784924632 --> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a80075710 con 0x7f6a80072c40 2024-01-21T03:20:04.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.158+0000 7f6a997fa640 20 client.4277.objecter dump_active .. 0 homeless 2024-01-21T03:20:04.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.159+0000 7f6aa103f640 1 --2- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] conn(0x7f6a80072c40 0x7f6a80075060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.161+0000 7f6aa103f640 10 monclient: get_auth_request con 0x7f6a80072c40 auth_method 0 2024-01-21T03:20:04.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.164+0000 7f6a997fa640 10 client.4277.objecter ms_handle_connect 0x7f6a80072c40 2024-01-21T03:20:04.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.164+0000 7f6aa103f640 1 --2- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] conn(0x7f6a80072c40 0x7f6a80075060 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.199+0000 7f3edeffd640 1 -- 172.21.15.7:0/1828190540 <== osd.4 v2:172.21.15.177:6808/3975537014 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3ec8075740 con 0x7f3ec8072c70 2024-01-21T03:20:04.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.199+0000 7f3edeffd640 10 client.4318.objecter ms_dispatch 0x7f3ee8000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:20:04.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.199+0000 7f3edeffd640 10 client.4318.objecter _finish_command 1 = system:0 2024-01-21T03:20:04.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.199+0000 7f3edeffd640 20 client.4318.objecter put_session s=0x7f3ec80725a0 osd=4 4 2024-01-21T03:20:04.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.199+0000 7f3edeffd640 15 client.4318.objecter _session_command_op_remove 4 1 2024-01-21T03:20:04.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.202+0000 7f5026ffd640 1 -- 172.21.15.7:0/1482777839 <== osd.6 v2:172.21.15.177:6800/1092304531 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5010079ba0 con 0x7f5010072cc0 2024-01-21T03:20:04.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.202+0000 7f5026ffd640 10 client.4281.objecter ms_dispatch 0x7f5030000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:20:04.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.202+0000 7f5026ffd640 10 client.4281.objecter _finish_command 1 = system:0 2024-01-21T03:20:04.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.202+0000 7f5026ffd640 20 client.4281.objecter put_session s=0x7f50100725f0 osd=6 4 2024-01-21T03:20:04.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.202+0000 7f5026ffd640 15 client.4281.objecter _session_command_op_remove 6 1 2024-01-21T03:20:04.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.210+0000 7f1894ff9640 1 -- 172.21.15.7:0/488701857 <== osd.3 v2:172.21.15.7:6802/1182044719 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f187c075710 con 0x7f187c072c40 2024-01-21T03:20:04.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.210+0000 7f1894ff9640 10 client.4312.objecter ms_dispatch 0x7f1898000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:20:04.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.210+0000 7f1894ff9640 10 client.4312.objecter _finish_command 1 = system:0 2024-01-21T03:20:04.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.210+0000 7f1894ff9640 20 client.4312.objecter put_session s=0x7f187c072570 osd=3 4 2024-01-21T03:20:04.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.210+0000 7f1894ff9640 15 client.4312.objecter _session_command_op_remove 3 1 2024-01-21T03:20:04.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.230+0000 7f3edeffd640 1 -- 172.21.15.7:0/1828190540 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ed80044f0 con 0x7f3ee80b2d60 2024-01-21T03:20:04.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f3edeffd640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-21T03:20:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f1894ff9640 1 -- 172.21.15.7:0/488701857 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f189806c4d0 con 0x7f1890035670 2024-01-21T03:20:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f1894ff9640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-21T03:20:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f3eece00640 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T03:20:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f3eece00640 15 client.4318.objecter _session_command_op_assign -1 2 2024-01-21T03:20:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f3eece00640 20 client.4318.objecter _get_session s=0x7f3ec80725a0 osd=4 3 2024-01-21T03:20:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f3eece00640 20 client.4318.objecter put_session s=0x7f3ec80725a0 osd=4 3 2024-01-21T03:20:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f3eece00640 20 client.4318.objecter _get_session s=0x7f3ec80725a0 osd=4 3 2024-01-21T03:20:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f3eece00640 15 client.4318.objecter _session_command_op_remove -1 2 2024-01-21T03:20:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f3eece00640 20 client.4318.objecter get_session s=0x7f3ec80725a0 osd=4 3 2024-01-21T03:20:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f3eece00640 15 client.4318.objecter _session_command_op_assign 4 2 2024-01-21T03:20:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f3eece00640 20 client.4318.objecter put_session s=0x7f3ec80725a0 osd=4 4 2024-01-21T03:20:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f3eece00640 10 client.4318.objecter _send_command 2 2024-01-21T03:20:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.231+0000 7f3eece00640 1 -- 172.21.15.7:0/1828190540 --> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3ee80728b0 con 0x7f3ec8072c70 2024-01-21T03:20:04.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.234+0000 7f50349e2640 10 client.4281.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T03:20:04.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.234+0000 7f50349e2640 15 client.4281.objecter _session_command_op_assign -1 2 2024-01-21T03:20:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.234+0000 7f50349e2640 20 client.4281.objecter _get_session s=0x7f50100725f0 osd=6 3 2024-01-21T03:20:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.234+0000 7f50349e2640 20 client.4281.objecter put_session s=0x7f50100725f0 osd=6 3 2024-01-21T03:20:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.234+0000 7f50349e2640 20 client.4281.objecter _get_session s=0x7f50100725f0 osd=6 3 2024-01-21T03:20:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.234+0000 7f50349e2640 15 client.4281.objecter _session_command_op_remove -1 2 2024-01-21T03:20:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.234+0000 7f50349e2640 20 client.4281.objecter get_session s=0x7f50100725f0 osd=6 3 2024-01-21T03:20:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.234+0000 7f50349e2640 15 client.4281.objecter _session_command_op_assign 6 2 2024-01-21T03:20:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.234+0000 7f50349e2640 20 client.4281.objecter put_session s=0x7f50100725f0 osd=6 4 2024-01-21T03:20:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.234+0000 7f50349e2640 10 client.4281.objecter _send_command 2 2024-01-21T03:20:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.234+0000 7f50349e2640 1 -- 172.21.15.7:0/1482777839 --> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f50300a8eb0 con 0x7f5010072cc0 2024-01-21T03:20:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f189d62e640 10 client.4312.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T03:20:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f189d62e640 15 client.4312.objecter _session_command_op_assign -1 2 2024-01-21T03:20:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f189d62e640 20 client.4312.objecter _get_session s=0x7f187c072570 osd=3 3 2024-01-21T03:20:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f189d62e640 20 client.4312.objecter put_session s=0x7f187c072570 osd=3 3 2024-01-21T03:20:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f189d62e640 20 client.4312.objecter _get_session s=0x7f187c072570 osd=3 3 2024-01-21T03:20:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f189d62e640 15 client.4312.objecter _session_command_op_remove -1 2 2024-01-21T03:20:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f189d62e640 20 client.4312.objecter get_session s=0x7f187c072570 osd=3 3 2024-01-21T03:20:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f189d62e640 15 client.4312.objecter _session_command_op_assign 3 2 2024-01-21T03:20:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f189d62e640 20 client.4312.objecter put_session s=0x7f187c072570 osd=3 4 2024-01-21T03:20:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f189d62e640 10 client.4312.objecter _send_command 2 2024-01-21T03:20:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f189d62e640 1 -- 172.21.15.7:0/488701857 --> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f18900bd760 con 0x7f187c072c40 2024-01-21T03:20:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f3edeffd640 1 -- 172.21.15.7:0/1828190540 <== osd.4 v2:172.21.15.177:6808/3975537014 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3ee0009ab0 con 0x7f3ec8072c70 2024-01-21T03:20:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f3edeffd640 10 client.4318.objecter ms_dispatch 0x7f3ee8000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:20:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f3edeffd640 10 client.4318.objecter _finish_command 2 = system:0 2024-01-21T03:20:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f3edeffd640 20 client.4318.objecter put_session s=0x7f3ec80725a0 osd=4 4 2024-01-21T03:20:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.238+0000 7f3edeffd640 15 client.4318.objecter _session_command_op_remove 4 2 2024-01-21T03:20:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 10 librados: watch_flush enter 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 10 librados: watch_flush exit 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 10 client.4318.objecter close_session for osd.4 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 1 -- 172.21.15.7:0/1828190540 >> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] conn(0x7f3ec8072c70 msgr2=0x7f3ec8075090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 1 --2- 172.21.15.7:0/1828190540 >> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] conn(0x7f3ec8072c70 0x7f3ec8075090 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 20 client.4318.objecter put_session s=0x7f3ec80725a0 osd=4 1 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 20 client.4318.objecter shutdown clearing up homeless session... 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 10 client.4318.objecter successfully canceled tick 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 1 -- 172.21.15.7:0/1828190540 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f3ec806c890 msgr2=0x7f3ec806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 1 --2- 172.21.15.7:0/1828190540 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f3ec806c890 0x7f3ec806ed50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0038000 tx=0x7f3ed003d040 comp rx=0 tx=0).stop 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 10 monclient: shutdown 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 1 -- 172.21.15.7:0/1828190540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ee80b2d60 msgr2=0x7f3ee80b1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 1 --2- 172.21.15.7:0/1828190540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ee80b2d60 0x7f3ee80b1fc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800c850 tx=0x7f3ed800cd10 comp rx=0 tx=0).stop 2024-01-21T03:20:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3ee6d9d640 1 -- 172.21.15.7:0/1828190540 reap_dead start 2024-01-21T03:20:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 1 -- 172.21.15.7:0/1828190540 shutdown_connections 2024-01-21T03:20:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 1 -- 172.21.15.7:0/1828190540 >> 172.21.15.7:0/1828190540 conn(0x7f3ee806aa50 msgr2=0x7f3ee8073df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 1 -- 172.21.15.7:0/1828190540 shutdown_connections 2024-01-21T03:20:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 1 -- 172.21.15.7:0/1828190540 wait complete. 2024-01-21T03:20:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.239+0000 7f3eece00640 1 librados: shutdown 2024-01-21T03:20:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.241+0000 7f8257fff640 1 -- 172.21.15.7:0/2865264600 <== osd.0 v2:172.21.15.7:6818/2019292672 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f825c075790 con 0x7f825c072cc0 2024-01-21T03:20:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.241+0000 7f8257fff640 10 client.4274.objecter ms_dispatch 0x7f8270000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:20:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.241+0000 7f8257fff640 10 client.4274.objecter _finish_command 1 = system:0 2024-01-21T03:20:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.241+0000 7f8257fff640 20 client.4274.objecter put_session s=0x7f825c0725f0 osd=0 4 2024-01-21T03:20:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.241+0000 7f8257fff640 15 client.4274.objecter _session_command_op_remove 0 1 2024-01-21T03:20:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.245+0000 7f5026ffd640 1 -- 172.21.15.7:0/1482777839 <== osd.6 v2:172.21.15.177:6800/1092304531 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f502802c8f0 con 0x7f5010072cc0 2024-01-21T03:20:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.245+0000 7f5026ffd640 10 client.4281.objecter ms_dispatch 0x7f5030000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:20:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.245+0000 7f5026ffd640 10 client.4281.objecter _finish_command 2 = system:0 2024-01-21T03:20:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.245+0000 7f5026ffd640 20 client.4281.objecter put_session s=0x7f50100725f0 osd=6 4 2024-01-21T03:20:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.245+0000 7f5026ffd640 15 client.4281.objecter _session_command_op_remove 6 2 2024-01-21T03:20:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.246+0000 7f6a997fa640 1 -- 172.21.15.7:0/2784924632 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a9c0ce630 con 0x7f6a94038db0 2024-01-21T03:20:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.246+0000 7f6a997fa640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-21T03:20:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f8257fff640 1 -- 172.21.15.7:0/2865264600 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8264077410 con 0x7f8270071910 2024-01-21T03:20:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f8257fff640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-21T03:20:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 10 librados: watch_flush enter 2024-01-21T03:20:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 10 librados: watch_flush exit 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 10 client.4281.objecter close_session for osd.6 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 1 -- 172.21.15.7:0/1482777839 >> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] conn(0x7f5010072cc0 msgr2=0x7f50100750e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 1 --2- 172.21.15.7:0/1482777839 >> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] conn(0x7f5010072cc0 0x7f50100750e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 20 client.4281.objecter put_session s=0x7f50100725f0 osd=6 1 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 20 client.4281.objecter shutdown clearing up homeless session... 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 10 client.4281.objecter successfully canceled tick 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 1 -- 172.21.15.7:0/1482777839 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f501006c8e0 msgr2=0x7f501006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 1 --2- 172.21.15.7:0/1482777839 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f501006c8e0 0x7f501006eda0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5030142ea0 tx=0x7f5018006c90 comp rx=0 tx=0).stop 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 10 monclient: shutdown 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 1 -- 172.21.15.7:0/1482777839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5030070a10 msgr2=0x7f50301d8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f50349e2640 1 --2- 172.21.15.7:0/1482777839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5030070a10 0x7f50301d8900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f502000d800 tx=0x7f502000dcc0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.247+0000 7f502ed9d640 1 -- 172.21.15.7:0/1482777839 reap_dead start 2024-01-21T03:20:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.248+0000 7f50349e2640 1 -- 172.21.15.7:0/1482777839 shutdown_connections 2024-01-21T03:20:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.248+0000 7f50349e2640 1 -- 172.21.15.7:0/1482777839 >> 172.21.15.7:0/1482777839 conn(0x7f5030066f50 msgr2=0x7f503013ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.250+0000 7f50349e2640 1 -- 172.21.15.7:0/1482777839 shutdown_connections 2024-01-21T03:20:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.250+0000 7f50349e2640 1 -- 172.21.15.7:0/1482777839 wait complete. 2024-01-21T03:20:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.250+0000 7f50349e2640 1 librados: shutdown 2024-01-21T03:20:04.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.255+0000 7f1894ff9640 1 -- 172.21.15.7:0/488701857 <== osd.3 v2:172.21.15.7:6802/1182044719 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f18900bd760 con 0x7f187c072c40 2024-01-21T03:20:04.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.255+0000 7f1894ff9640 10 client.4312.objecter ms_dispatch 0x7f1898000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:20:04.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.255+0000 7f1894ff9640 10 client.4312.objecter _finish_command 2 = system:0 2024-01-21T03:20:04.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.255+0000 7f1894ff9640 20 client.4312.objecter put_session s=0x7f187c072570 osd=3 4 2024-01-21T03:20:04.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.255+0000 7f1894ff9640 15 client.4312.objecter _session_command_op_remove 3 2 2024-01-21T03:20:04.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.255+0000 7f189d62e640 10 librados: watch_flush enter 2024-01-21T03:20:04.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.255+0000 7f189d62e640 10 librados: watch_flush exit 2024-01-21T03:20:04.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.255+0000 7f189d62e640 10 client.4312.objecter close_session for osd.3 2024-01-21T03:20:04.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.255+0000 7f189d62e640 1 -- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] conn(0x7f187c072c40 msgr2=0x7f187c075060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.255+0000 7f189d62e640 1 --2- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] conn(0x7f187c072c40 0x7f187c075060 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.255+0000 7f189d62e640 20 client.4312.objecter put_session s=0x7f187c072570 osd=3 1 2024-01-21T03:20:04.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.255+0000 7f189d62e640 20 client.4312.objecter shutdown clearing up homeless session... 2024-01-21T03:20:04.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.255+0000 7f189d62e640 10 client.4312.objecter successfully canceled tick 2024-01-21T03:20:04.243 INFO:teuthology.orchestra.run.smithi007.stdout:60129542152 2024-01-21T03:20:04.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-21T03:20:04.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.256+0000 7f189d62e640 1 -- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f187c06c860 msgr2=0x7f187c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.256+0000 7f189d62e640 1 --2- 172.21.15.7:0/488701857 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f187c06c860 0x7f187c06ed20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f18900cc1e0 tx=0x7f18840079b0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.256+0000 7f189d62e640 10 monclient: shutdown 2024-01-21T03:20:04.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.256+0000 7f189d62e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.256+0000 7f189d62e640 1 -- 172.21.15.7:0/488701857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1890035670 msgr2=0x7f1890033c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.256+0000 7f189d62e640 1 --2- 172.21.15.7:0/488701857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1890035670 0x7f1890033c80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f189804ee90 tx=0x7f189806ca20 comp rx=0 tx=0).stop 2024-01-21T03:20:04.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.256+0000 7f189ce2d640 1 -- 172.21.15.7:0/488701857 reap_dead start 2024-01-21T03:20:04.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.256+0000 7f189d62e640 1 -- 172.21.15.7:0/488701857 shutdown_connections 2024-01-21T03:20:04.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.256+0000 7f189d62e640 1 -- 172.21.15.7:0/488701857 >> 172.21.15.7:0/488701857 conn(0x7f1890006600 msgr2=0x7f1890034ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.256+0000 7f189d62e640 1 -- 172.21.15.7:0/488701857 shutdown_connections 2024-01-21T03:20:04.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.256+0000 7f189d62e640 1 -- 172.21.15.7:0/488701857 wait complete. 2024-01-21T03:20:04.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.256+0000 7f189d62e640 1 librados: shutdown 2024-01-21T03:20:04.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.260+0000 7f6a997fa640 1 -- 172.21.15.7:0/2784924632 <== osd.1 v2:172.21.15.7:6810/2176696455 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6a80075710 con 0x7f6a80072c40 2024-01-21T03:20:04.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.260+0000 7f6a997fa640 10 client.4277.objecter ms_dispatch 0x7f6a9c000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:20:04.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.260+0000 7f6a997fa640 10 client.4277.objecter _finish_command 1 = system:0 2024-01-21T03:20:04.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.260+0000 7f6a997fa640 20 client.4277.objecter put_session s=0x7f6a80072570 osd=1 4 2024-01-21T03:20:04.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.260+0000 7f6a997fa640 15 client.4277.objecter _session_command_op_remove 1 1 2024-01-21T03:20:04.251 INFO:teuthology.orchestra.run.smithi007.stdout:77309411337 2024-01-21T03:20:04.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-21T03:20:04.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.268+0000 7f8275c46640 10 client.4274.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T03:20:04.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.268+0000 7f8275c46640 15 client.4274.objecter _session_command_op_assign -1 2 2024-01-21T03:20:04.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.268+0000 7f8275c46640 20 client.4274.objecter _get_session s=0x7f825c0725f0 osd=0 3 2024-01-21T03:20:04.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.268+0000 7f8275c46640 20 client.4274.objecter put_session s=0x7f825c0725f0 osd=0 3 2024-01-21T03:20:04.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.268+0000 7f8275c46640 20 client.4274.objecter _get_session s=0x7f825c0725f0 osd=0 3 2024-01-21T03:20:04.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.268+0000 7f8275c46640 15 client.4274.objecter _session_command_op_remove -1 2 2024-01-21T03:20:04.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.268+0000 7f8275c46640 20 client.4274.objecter get_session s=0x7f825c0725f0 osd=0 3 2024-01-21T03:20:04.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.268+0000 7f8275c46640 15 client.4274.objecter _session_command_op_assign 0 2 2024-01-21T03:20:04.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.268+0000 7f8275c46640 20 client.4274.objecter put_session s=0x7f825c0725f0 osd=0 4 2024-01-21T03:20:04.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.268+0000 7f8275c46640 10 client.4274.objecter _send_command 2 2024-01-21T03:20:04.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.268+0000 7f8275c46640 1 -- 172.21.15.7:0/2865264600 --> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8270070c70 con 0x7f825c072cc0 2024-01-21T03:20:04.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00edffb640 1 -- 172.21.15.7:0/3891581075 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f000cba0 con 0x7f00f8142b80 2024-01-21T03:20:04.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00edffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:04.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00edffb640 10 monclient: dump: 2024-01-21T03:20:04.256 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.256 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.256 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.256 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.256 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.256 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.256 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.256 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.256 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.256 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00edffb640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00edffb640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00edffb640 1 -- 172.21.15.7:0/3891581075 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f000cd00 con 0x7f00f8142b80 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00edffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00edffb640 1 -- 172.21.15.7:0/3891581075 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f0014680 con 0x7f00f8142b80 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00edffb640 10 monclient: dump: 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.257 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00ff7e2640 5 monclient: authenticate success, global_id 4287 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00ff7e2640 10 monclient: _renew_subs 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00ff7e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00ff7e2640 1 -- 172.21.15.7:0/3891581075 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f8143ac0 con 0x7f00f8142b80 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00ff7e2640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00ff7e2640 10 monclient: _renew_subs 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00ef7fe640 4 set_mon_vals no callback set 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00ff7e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00ff7e2640 1 -- 172.21.15.7:0/3891581075 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f81d7ae0 con 0x7f00f8142b80 2024-01-21T03:20:04.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.269+0000 7f00ff7e2640 1 librados: init done 2024-01-21T03:20:04.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.270+0000 7f00ff7e2640 10 client.4287.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.270+0000 7f00ff7e2640 15 client.4287.objecter _session_command_op_assign -1 1 2024-01-21T03:20:04.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.270+0000 7f00ff7e2640 20 client.4287.objecter _get_session osd=-1 returning homeless 2024-01-21T03:20:04.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.270+0000 7f00ff7e2640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.270+0000 7f00ff7e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.270+0000 7f00ff7e2640 1 -- 172.21.15.7:0/3891581075 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f00f813fc50 con 0x7f00f8142b80 2024-01-21T03:20:04.260 INFO:teuthology.orchestra.run.smithi007.stdout:94489280517 2024-01-21T03:20:04.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-21T03:20:04.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.281+0000 7f6aa1840640 10 client.4277.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T03:20:04.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.281+0000 7f6aa1840640 15 client.4277.objecter _session_command_op_assign -1 2 2024-01-21T03:20:04.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.281+0000 7f6aa1840640 20 client.4277.objecter _get_session s=0x7f6a80072570 osd=1 3 2024-01-21T03:20:04.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.281+0000 7f6aa1840640 20 client.4277.objecter put_session s=0x7f6a80072570 osd=1 3 2024-01-21T03:20:04.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.281+0000 7f6aa1840640 20 client.4277.objecter _get_session s=0x7f6a80072570 osd=1 3 2024-01-21T03:20:04.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.281+0000 7f6aa1840640 15 client.4277.objecter _session_command_op_remove -1 2 2024-01-21T03:20:04.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.281+0000 7f6aa1840640 20 client.4277.objecter get_session s=0x7f6a80072570 osd=1 3 2024-01-21T03:20:04.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.281+0000 7f6aa1840640 15 client.4277.objecter _session_command_op_assign 1 2 2024-01-21T03:20:04.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.281+0000 7f6aa1840640 20 client.4277.objecter put_session s=0x7f6a80072570 osd=1 4 2024-01-21T03:20:04.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.281+0000 7f6aa1840640 10 client.4277.objecter _send_command 2 2024-01-21T03:20:04.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.281+0000 7f6aa1840640 1 -- 172.21.15.7:0/2784924632 --> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6a940366d0 con 0x7f6a80072c40 2024-01-21T03:20:04.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.293+0000 7f8257fff640 1 -- 172.21.15.7:0/2865264600 <== osd.0 v2:172.21.15.7:6818/2019292672 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8270070c70 con 0x7f825c072cc0 2024-01-21T03:20:04.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.293+0000 7f8257fff640 10 client.4274.objecter ms_dispatch 0x7f8270000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:20:04.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.293+0000 7f8257fff640 10 client.4274.objecter _finish_command 2 = system:0 2024-01-21T03:20:04.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.293+0000 7f8257fff640 20 client.4274.objecter put_session s=0x7f825c0725f0 osd=0 4 2024-01-21T03:20:04.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.293+0000 7f8257fff640 15 client.4274.objecter _session_command_op_remove 0 2 2024-01-21T03:20:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.293+0000 7f8275c46640 10 librados: watch_flush enter 2024-01-21T03:20:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.293+0000 7f8275c46640 10 librados: watch_flush exit 2024-01-21T03:20:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 10 client.4274.objecter close_session for osd.0 2024-01-21T03:20:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 1 -- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] conn(0x7f825c072cc0 msgr2=0x7f825c0750e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 1 --2- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] conn(0x7f825c072cc0 0x7f825c0750e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 20 client.4274.objecter put_session s=0x7f825c0725f0 osd=0 1 2024-01-21T03:20:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 20 client.4274.objecter shutdown clearing up homeless session... 2024-01-21T03:20:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 10 client.4274.objecter successfully canceled tick 2024-01-21T03:20:04.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 1 -- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f825c06c8e0 msgr2=0x7f825c06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 1 --2- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f825c06c8e0 0x7f825c06eda0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8260001560 tx=0x7f8260007990 comp rx=0 tx=0).stop 2024-01-21T03:20:04.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 10 monclient: shutdown 2024-01-21T03:20:04.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 1 -- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8270071910 msgr2=0x7f82700ac190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 1 --2- 172.21.15.7:0/2865264600 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8270071910 0x7f82700ac190 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f826402b950 tx=0x7f82640046c0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f826ffff640 1 -- 172.21.15.7:0/2865264600 reap_dead start 2024-01-21T03:20:04.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 1 -- 172.21.15.7:0/2865264600 shutdown_connections 2024-01-21T03:20:04.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 1 -- 172.21.15.7:0/2865264600 >> 172.21.15.7:0/2865264600 conn(0x7f8270066d30 msgr2=0x7f82700b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 1 -- 172.21.15.7:0/2865264600 shutdown_connections 2024-01-21T03:20:04.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.294+0000 7f8275c46640 1 -- 172.21.15.7:0/2865264600 wait complete. 2024-01-21T03:20:04.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.295+0000 7f8275c46640 1 librados: shutdown 2024-01-21T03:20:04.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.295+0000 7f40d9ffb640 1 -- 172.21.15.7:0/826364513 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f40d4003bc0 con 0x7f40e4070b50 2024-01-21T03:20:04.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.295+0000 7f40d9ffb640 1 --2- 172.21.15.7:0/826364513 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f40c806c860 0x7f40c806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.296+0000 7f40e8e24640 1 --2- 172.21.15.7:0/826364513 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f40c806c860 0x7f40c806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.296+0000 7f40e8e24640 10 monclient: get_auth_request con 0x7f40c806c860 auth_method 0 2024-01-21T03:20:04.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.296+0000 7f40e8e24640 1 --2- 172.21.15.7:0/826364513 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f40c806c860 0x7f40c806ed20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0012e0 tx=0x7f40dc001650 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.296+0000 7f40d9ffb640 10 client.4284.objecter ms_handle_connect 0x7f40c806c860 2024-01-21T03:20:04.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 1 -- 172.21.15.7:0/826364513 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f40d4032390 con 0x7f40e4070b50 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 10 client.4284.objecter ms_dispatch 0x7f40e4000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 3 client.4284.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 3 client.4284.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 10 client.4284.objecter checking command 1 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 1 --2- 172.21.15.7:0/826364513 >> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] conn(0x7f40c8073560 0x7f40c8075a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 20 client.4284.objecter _get_session s=0x7f40c8072e90 osd=5 3 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 20 client.4284.objecter put_session s=0x7f40c8072e90 osd=5 3 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 15 client.4284.objecter _session_command_op_remove -1 1 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 20 client.4284.objecter _get_session s=0x7f40c8072e90 osd=5 3 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 20 client.4284.objecter get_session s=0x7f40c8072e90 osd=5 3 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 15 client.4284.objecter _session_command_op_assign 5 1 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 20 client.4284.objecter put_session s=0x7f40c8072e90 osd=5 4 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 10 client.4284.objecter _send_command 1 2024-01-21T03:20:04.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 1 -- 172.21.15.7:0/826364513 --> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40c80760d0 con 0x7f40c8073560 2024-01-21T03:20:04.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.303+0000 7f40d9ffb640 20 client.4284.objecter dump_active .. 0 homeless 2024-01-21T03:20:04.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.304+0000 7f40d9ffb640 1 -- 172.21.15.7:0/826364513 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40d402f020 con 0x7f40e4070b50 2024-01-21T03:20:04.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.304+0000 7f40d9ffb640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-21T03:20:04.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.304+0000 7f40e9e26640 1 --2- 172.21.15.7:0/826364513 >> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] conn(0x7f40c8073560 0x7f40c8075a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.305+0000 7f40e9e26640 10 monclient: get_auth_request con 0x7f40c8073560 auth_method 0 2024-01-21T03:20:04.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.306+0000 7f6a997fa640 1 -- 172.21.15.7:0/2784924632 <== osd.1 v2:172.21.15.7:6810/2176696455 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6a9000b920 con 0x7f6a80072c40 2024-01-21T03:20:04.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.306+0000 7f6a997fa640 10 client.4277.objecter ms_dispatch 0x7f6a9c000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:20:04.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.306+0000 7f6a997fa640 10 client.4277.objecter _finish_command 2 = system:0 2024-01-21T03:20:04.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.306+0000 7f6a997fa640 20 client.4277.objecter put_session s=0x7f6a80072570 osd=1 4 2024-01-21T03:20:04.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.306+0000 7f6a997fa640 15 client.4277.objecter _session_command_op_remove 1 2 2024-01-21T03:20:04.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 10 librados: watch_flush enter 2024-01-21T03:20:04.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 10 librados: watch_flush exit 2024-01-21T03:20:04.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 10 client.4277.objecter close_session for osd.1 2024-01-21T03:20:04.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 1 -- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] conn(0x7f6a80072c40 msgr2=0x7f6a80075060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 1 --2- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] conn(0x7f6a80072c40 0x7f6a80075060 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 20 client.4277.objecter put_session s=0x7f6a80072570 osd=1 1 2024-01-21T03:20:04.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 20 client.4277.objecter shutdown clearing up homeless session... 2024-01-21T03:20:04.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 10 client.4277.objecter successfully canceled tick 2024-01-21T03:20:04.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 1 -- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6a8006c860 msgr2=0x7f6a8006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 1 --2- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6a8006c860 0x7f6a8006ed20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6a88036000 tx=0x7f6a88030000 comp rx=0 tx=0).stop 2024-01-21T03:20:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 10 monclient: shutdown 2024-01-21T03:20:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 1 -- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a94038db0 msgr2=0x7f6a940c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 1 --2- 172.21.15.7:0/2784924632 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6a94038db0 0x7f6a940c9cb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c075e20 tx=0x7f6a9c067a90 comp rx=0 tx=0).stop 2024-01-21T03:20:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa103f640 1 -- 172.21.15.7:0/2784924632 reap_dead start 2024-01-21T03:20:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 1 -- 172.21.15.7:0/2784924632 shutdown_connections 2024-01-21T03:20:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.307+0000 7f6aa1840640 1 -- 172.21.15.7:0/2784924632 >> 172.21.15.7:0/2784924632 conn(0x7f6a94006610 msgr2=0x7f6a940c4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.308+0000 7f40e9e26640 1 --2- 172.21.15.7:0/826364513 >> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] conn(0x7f40c8073560 0x7f40c8075a00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.308+0000 7f40d9ffb640 10 client.4284.objecter ms_handle_connect 0x7f40c8073560 2024-01-21T03:20:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.308+0000 7f6aa1840640 1 -- 172.21.15.7:0/2784924632 shutdown_connections 2024-01-21T03:20:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.309+0000 7f6aa1840640 1 -- 172.21.15.7:0/2784924632 wait complete. 2024-01-21T03:20:04.297 INFO:teuthology.orchestra.run.smithi007.stdout:90194313225 2024-01-21T03:20:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.311+0000 7f6aa1840640 1 librados: shutdown 2024-01-21T03:20:04.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-21T03:20:04.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.320+0000 7fc3a77fe640 1 -- 172.21.15.7:0/648551170 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc3c0090070 con 0x7fc3b802fc30 2024-01-21T03:20:04.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.321+0000 7fc3a77fe640 1 --2- 172.21.15.7:0/648551170 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fc3ac06c8b0 0x7fc3ac06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.321+0000 7fc3beffd640 1 --2- 172.21.15.7:0/648551170 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fc3ac06c8b0 0x7fc3ac06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.321+0000 7fc3beffd640 10 monclient: get_auth_request con 0x7fc3ac06c8b0 auth_method 0 2024-01-21T03:20:04.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.322+0000 7fc3beffd640 1 --2- 172.21.15.7:0/648551170 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fc3ac06c8b0 0x7fc3ac06ed70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0036000 tx=0x7fc3b003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.322+0000 7fc3a77fe640 10 client.4290.objecter ms_handle_connect 0x7fc3ac06c8b0 2024-01-21T03:20:04.313 INFO:teuthology.orchestra.run.smithi007.stdout:90194313224 2024-01-21T03:20:04.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-21T03:20:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 1 -- 172.21.15.7:0/648551170 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc3c00d2810 con 0x7fc3b802fc30 2024-01-21T03:20:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 10 client.4290.objecter ms_dispatch 0x7fc3c0000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 3 client.4290.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 3 client.4290.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 10 client.4290.objecter checking command 1 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 1 --2- 172.21.15.7:0/648551170 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x7fc3ac072c90 0x7fc3ac0750b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 20 client.4290.objecter _get_session s=0x7fc3ac0725c0 osd=7 3 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 20 client.4290.objecter put_session s=0x7fc3ac0725c0 osd=7 3 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 15 client.4290.objecter _session_command_op_remove -1 1 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 20 client.4290.objecter _get_session s=0x7fc3ac0725c0 osd=7 3 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 20 client.4290.objecter get_session s=0x7fc3ac0725c0 osd=7 3 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 15 client.4290.objecter _session_command_op_assign 7 1 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 20 client.4290.objecter put_session s=0x7fc3ac0725c0 osd=7 4 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 10 client.4290.objecter _send_command 1 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 1 -- 172.21.15.7:0/648551170 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3ac075760 con 0x7fc3ac072c90 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 20 client.4290.objecter dump_active .. 0 homeless 2024-01-21T03:20:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 1 -- 172.21.15.7:0/648551170 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3c00ce450 con 0x7fc3b802fc30 2024-01-21T03:20:04.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.335+0000 7fc3a77fe640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-21T03:20:04.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.336+0000 7fc3be7fc640 1 --2- 172.21.15.7:0/648551170 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x7fc3ac072c90 0x7fc3ac0750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.336+0000 7fc3be7fc640 10 monclient: get_auth_request con 0x7fc3ac072c90 auth_method 0 2024-01-21T03:20:04.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.340+0000 7fc3be7fc640 1 --2- 172.21.15.7:0/648551170 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x7fc3ac072c90 0x7fc3ac0750b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.340+0000 7fc3a77fe640 10 client.4290.objecter ms_handle_connect 0x7fc3ac072c90 2024-01-21T03:20:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.350+0000 7f40d9ffb640 1 -- 172.21.15.7:0/826364513 <== osd.5 v2:172.21.15.177:6816/2318383718 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f40c80760d0 con 0x7f40c8073560 2024-01-21T03:20:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.350+0000 7f40d9ffb640 10 client.4284.objecter ms_dispatch 0x7f40e4000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:20:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.350+0000 7f40d9ffb640 10 client.4284.objecter _finish_command 1 = system:0 2024-01-21T03:20:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.350+0000 7f40d9ffb640 20 client.4284.objecter put_session s=0x7f40c8072e90 osd=5 4 2024-01-21T03:20:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.350+0000 7f40d9ffb640 15 client.4284.objecter _session_command_op_remove 5 1 2024-01-21T03:20:04.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.353+0000 7f00edffb640 1 -- 172.21.15.7:0/3891581075 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f00f001d3b0 con 0x7f00f8142b80 2024-01-21T03:20:04.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.354+0000 7f00edffb640 1 --2- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f00dc06c8b0 0x7f00dc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.354+0000 7f00fd57e640 1 --2- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f00dc06c8b0 0x7f00dc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.354+0000 7f00fd57e640 10 monclient: get_auth_request con 0x7f00dc06c8b0 auth_method 0 2024-01-21T03:20:04.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.355+0000 7f00fd57e640 1 --2- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f00dc06c8b0 0x7f00dc06ed70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f00f4035000 tx=0x7f00f4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.355+0000 7f00edffb640 10 client.4287.objecter ms_handle_connect 0x7f00dc06c8b0 2024-01-21T03:20:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.382+0000 7f40ea627640 10 client.4284.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T03:20:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.382+0000 7f40ea627640 15 client.4284.objecter _session_command_op_assign -1 2 2024-01-21T03:20:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.382+0000 7f40ea627640 20 client.4284.objecter _get_session s=0x7f40c8072e90 osd=5 3 2024-01-21T03:20:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.382+0000 7f40ea627640 20 client.4284.objecter put_session s=0x7f40c8072e90 osd=5 3 2024-01-21T03:20:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.382+0000 7f40ea627640 20 client.4284.objecter _get_session s=0x7f40c8072e90 osd=5 3 2024-01-21T03:20:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.382+0000 7f40ea627640 15 client.4284.objecter _session_command_op_remove -1 2 2024-01-21T03:20:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.382+0000 7f40ea627640 20 client.4284.objecter get_session s=0x7f40c8072e90 osd=5 3 2024-01-21T03:20:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.382+0000 7f40ea627640 15 client.4284.objecter _session_command_op_assign 5 2 2024-01-21T03:20:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.382+0000 7f40ea627640 20 client.4284.objecter put_session s=0x7f40c8072e90 osd=5 4 2024-01-21T03:20:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.382+0000 7f40ea627640 10 client.4284.objecter _send_command 2 2024-01-21T03:20:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.382+0000 7f40ea627640 1 -- 172.21.15.7:0/826364513 --> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f40e406e9e0 con 0x7f40c8073560 2024-01-21T03:20:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.385+0000 7fc3a77fe640 1 -- 172.21.15.7:0/648551170 <== osd.7 v2:172.21.15.177:6823/4069998673 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc3ac075760 con 0x7fc3ac072c90 2024-01-21T03:20:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.385+0000 7fc3a77fe640 10 client.4290.objecter ms_dispatch 0x7fc3c0000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:20:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.385+0000 7fc3a77fe640 10 client.4290.objecter _finish_command 1 = system:0 2024-01-21T03:20:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.385+0000 7fc3a77fe640 20 client.4290.objecter put_session s=0x7fc3ac0725c0 osd=7 4 2024-01-21T03:20:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.385+0000 7fc3a77fe640 15 client.4290.objecter _session_command_op_remove 7 1 2024-01-21T03:20:04.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.393+0000 7f40d9ffb640 1 -- 172.21.15.7:0/826364513 <== osd.5 v2:172.21.15.177:6816/2318383718 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f40e406e9e0 con 0x7f40c8073560 2024-01-21T03:20:04.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.393+0000 7f40d9ffb640 10 client.4284.objecter ms_dispatch 0x7f40e4000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:20:04.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.393+0000 7f40d9ffb640 10 client.4284.objecter _finish_command 2 = system:0 2024-01-21T03:20:04.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.393+0000 7f40d9ffb640 20 client.4284.objecter put_session s=0x7f40c8072e90 osd=5 4 2024-01-21T03:20:04.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.393+0000 7f40d9ffb640 15 client.4284.objecter _session_command_op_remove 5 2 2024-01-21T03:20:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 10 librados: watch_flush enter 2024-01-21T03:20:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 10 librados: watch_flush exit 2024-01-21T03:20:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 10 client.4284.objecter close_session for osd.5 2024-01-21T03:20:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 1 -- 172.21.15.7:0/826364513 >> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] conn(0x7f40c8073560 msgr2=0x7f40c8075a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 1 --2- 172.21.15.7:0/826364513 >> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] conn(0x7f40c8073560 0x7f40c8075a00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 20 client.4284.objecter put_session s=0x7f40c8072e90 osd=5 1 2024-01-21T03:20:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 20 client.4284.objecter shutdown clearing up homeless session... 2024-01-21T03:20:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 10 client.4284.objecter successfully canceled tick 2024-01-21T03:20:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 1 -- 172.21.15.7:0/826364513 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f40c806c860 msgr2=0x7f40c806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 1 --2- 172.21.15.7:0/826364513 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f40c806c860 0x7f40c806ed20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0012e0 tx=0x7f40dc001650 comp rx=0 tx=0).stop 2024-01-21T03:20:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 10 monclient: shutdown 2024-01-21T03:20:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 1 -- 172.21.15.7:0/826364513 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4070b50 msgr2=0x7f40e413c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40ea627640 1 --2- 172.21.15.7:0/826364513 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4070b50 0x7f40e413c580 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f40d402b7e0 tx=0x7f40d40046c0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.394+0000 7f40e9e26640 1 -- 172.21.15.7:0/826364513 reap_dead start 2024-01-21T03:20:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.395+0000 7f40ea627640 1 -- 172.21.15.7:0/826364513 shutdown_connections 2024-01-21T03:20:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.395+0000 7f40ea627640 1 -- 172.21.15.7:0/826364513 >> 172.21.15.7:0/826364513 conn(0x7f40e40b0040 msgr2=0x7f40e40aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.402+0000 7f40ea627640 1 -- 172.21.15.7:0/826364513 shutdown_connections 2024-01-21T03:20:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.402+0000 7f40ea627640 1 -- 172.21.15.7:0/826364513 wait complete. 2024-01-21T03:20:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.402+0000 7f40ea627640 1 librados: shutdown 2024-01-21T03:20:04.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.418+0000 7fc3a57fa640 10 client.4290.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T03:20:04.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.418+0000 7fc3a57fa640 15 client.4290.objecter _session_command_op_assign -1 2 2024-01-21T03:20:04.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.418+0000 7fc3a57fa640 20 client.4290.objecter _get_session s=0x7fc3ac0725c0 osd=7 3 2024-01-21T03:20:04.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.418+0000 7fc3a57fa640 20 client.4290.objecter put_session s=0x7fc3ac0725c0 osd=7 3 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.418+0000 7fc3a57fa640 20 client.4290.objecter _get_session s=0x7fc3ac0725c0 osd=7 3 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.418+0000 7fc3a57fa640 15 client.4290.objecter _session_command_op_remove -1 2 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.418+0000 7fc3a57fa640 20 client.4290.objecter get_session s=0x7fc3ac0725c0 osd=7 3 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.418+0000 7fc3a57fa640 15 client.4290.objecter _session_command_op_assign 7 2 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.418+0000 7fc3a57fa640 20 client.4290.objecter put_session s=0x7fc3ac0725c0 osd=7 4 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.418+0000 7fc3a57fa640 10 client.4290.objecter _send_command 2 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.418+0000 7fc3a57fa640 1 -- 172.21.15.7:0/648551170 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc398001150 con 0x7fc3ac072c90 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.411+0000 7f1689d3c640 10 monclient: get_monmap_and_config 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.411+0000 7f1689d3c640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.411+0000 7f1689d3c640 10 monclient: monmap: 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.412040+0000 2024-01-21T03:20:04.405 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.412040+0000 2024-01-21T03:20:04.406 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.406 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.406 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:04.406 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:04.406 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:04.406 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:04.406 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:04.406 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.419+0000 7f1689d3c640 1 Processor -- start 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.419+0000 7f1689d3c640 1 -- start start 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.419+0000 7f1689d3c640 10 monclient: init 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f168406f290 0x7f16840712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 10 monclient(hunting): picked mon.noname-a con 0x7f168406f290 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f168406f670 0x7f16840717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 10 monclient(hunting): picked mon.noname-c con 0x7f168406f670 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1684146200 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1684070bb0 con 0x7f1684146200 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1684070e90 con 0x7f168406f290 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1684071d30 con 0x7f168406f670 2024-01-21T03:20:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7f1689d3c640 10 monclient(hunting): authenticate will time out at 1350.736137s 2024-01-21T03:20:04.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7fabfc947640 10 monclient: get_monmap_and_config 2024-01-21T03:20:04.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7fabfc947640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.420+0000 7fabfc947640 10 monclient: monmap: 2024-01-21T03:20:04.408 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.421343+0000 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.421343+0000 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.421+0000 7fabfc947640 1 Processor -- start 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.421+0000 7fabfc947640 1 -- start start 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.421+0000 7fabfc947640 10 monclient: init 2024-01-21T03:20:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fabf0035240 0x7fabf002f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 10 monclient(hunting): picked mon.noname-c con 0x7fabf0035240 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fabf0035620 0x7fabf00c7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 10 monclient(hunting): picked mon.noname-a con 0x7fabf0035620 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabf00326d0 0x7fabf00349b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 10 monclient(hunting): picked mon.noname-b con 0x7fabf00326d0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf0031af0 con 0x7fabf0035620 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf0031d70 con 0x7fabf0035240 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00332c0 con 0x7fabf00326d0 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.422+0000 7fabfc947640 10 monclient(hunting): authenticate will time out at 1350.738551s 2024-01-21T03:20:04.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7f1688d3a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f168406f290 0x7f16840712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7f1688d3a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f168406f290 0x7f16840712b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44222/0 (socket says 172.21.15.7:44222) 2024-01-21T03:20:04.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7f1688d3a640 1 -- 172.21.15.7:0/1481036120 learned_addr learned my addr 172.21.15.7:0/1481036120 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7f1688d3a640 10 monclient(hunting): get_auth_request con 0x7f168406f290 auth_method 0 2024-01-21T03:20:04.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7f1688d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7f1688d3a640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7f1688d3a640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7fabf6ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fabf0035620 0x7fabf00c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7fabf6ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fabf0035620 0x7fabf00c7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44232/0 (socket says 172.21.15.7:44232) 2024-01-21T03:20:04.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7fabf6ffd640 1 -- 172.21.15.7:0/4292718646 learned_addr learned my addr 172.21.15.7:0/4292718646 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7fabf6ffd640 10 monclient(hunting): get_auth_request con 0x7fabf0035620 auth_method 0 2024-01-21T03:20:04.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7fabf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7fabf6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.425+0000 7fabf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.413 INFO:teuthology.orchestra.run.smithi007.stdout:77309411338 2024-01-21T03:20:04.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-21T03:20:04.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.426+0000 7f1688d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.426+0000 7f1688d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.426+0000 7f1688d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.428+0000 7fc3a77fe640 1 -- 172.21.15.7:0/648551170 <== osd.7 v2:172.21.15.177:6823/4069998673 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc398001150 con 0x7fc3ac072c90 2024-01-21T03:20:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.428+0000 7fc3a77fe640 10 client.4290.objecter ms_dispatch 0x7fc3c0000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:20:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.428+0000 7fc3a77fe640 10 client.4290.objecter _finish_command 2 = system:0 2024-01-21T03:20:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.428+0000 7fc3a77fe640 20 client.4290.objecter put_session s=0x7fc3ac0725c0 osd=7 4 2024-01-21T03:20:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.428+0000 7fc3a77fe640 15 client.4290.objecter _session_command_op_remove 7 2 2024-01-21T03:20:04.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.434+0000 7f1683fff640 1 --2- 172.21.15.7:0/1481036120 >> v2:172.21.15.177:3300/0 conn(0x7f168406f670 0x7f16840717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.434+0000 7f1683fff640 10 monclient(hunting): get_auth_request con 0x7f168406f670 auth_method 0 2024-01-21T03:20:04.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.434+0000 7f1683fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.434+0000 7f1683fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.434+0000 7f1683fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.434+0000 7fabf77fe640 1 --2- 172.21.15.7:0/4292718646 >> v2:172.21.15.177:3300/0 conn(0x7fabf0035240 0x7fabf002f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.434+0000 7fabf77fe640 10 monclient(hunting): get_auth_request con 0x7fabf0035240 auth_method 0 2024-01-21T03:20:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.434+0000 7fabf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.434+0000 7fabf77fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.434+0000 7fabf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 10 librados: watch_flush enter 2024-01-21T03:20:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 10 librados: watch_flush exit 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 10 client.4290.objecter close_session for osd.7 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 1 -- 172.21.15.7:0/648551170 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x7fc3ac072c90 msgr2=0x7fc3ac0750b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 1 --2- 172.21.15.7:0/648551170 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x7fc3ac072c90 0x7fc3ac0750b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 20 client.4290.objecter put_session s=0x7fc3ac0725c0 osd=7 1 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 20 client.4290.objecter shutdown clearing up homeless session... 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 10 client.4290.objecter successfully canceled tick 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 1 -- 172.21.15.7:0/648551170 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fc3ac06c8b0 msgr2=0x7fc3ac06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 1 --2- 172.21.15.7:0/648551170 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fc3ac06c8b0 0x7fc3ac06ed70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0036000 tx=0x7fc3b003d040 comp rx=0 tx=0).stop 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 10 monclient: shutdown 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 1 -- 172.21.15.7:0/648551170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b802fc30 msgr2=0x7fc3b80300e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 1 --2- 172.21.15.7:0/648551170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b802fc30 0x7fc3b80300e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0050150 tx=0x7fc3c0067a90 comp rx=0 tx=0).stop 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3bf7fe640 1 -- 172.21.15.7:0/648551170 reap_dead start 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 1 -- 172.21.15.7:0/648551170 shutdown_connections 2024-01-21T03:20:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.432+0000 7fc3a57fa640 1 -- 172.21.15.7:0/648551170 >> 172.21.15.7:0/648551170 conn(0x7fc3b8006640 msgr2=0x7fc3b8038520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.433+0000 7fc3a57fa640 1 -- 172.21.15.7:0/648551170 shutdown_connections 2024-01-21T03:20:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.433+0000 7fc3a57fa640 1 -- 172.21.15.7:0/648551170 wait complete. 2024-01-21T03:20:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.433+0000 7fc3a57fa640 1 librados: shutdown 2024-01-21T03:20:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.441+0000 7f1683fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.441+0000 7f1683fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.441+0000 7f1683fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.441+0000 7fabf77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.441+0000 7fabf77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.441+0000 7fabf77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.441+0000 7fabf7fff640 1 --2- 172.21.15.7:0/4292718646 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabf00326d0 0x7fabf00349b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.441+0000 7fabf7fff640 10 monclient(hunting): get_auth_request con 0x7fabf00326d0 auth_method 0 2024-01-21T03:20:04.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.441+0000 7fabf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.441+0000 7fabf7fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.441+0000 7fabf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.443+0000 7f1688d3a640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-01-21T03:20:04.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.443+0000 7f1688d3a640 1 -- 172.21.15.7:0/1481036120 >> v2:172.21.15.177:3300/0 conn(0x7f168406f670 msgr2=0x7f16840717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.443+0000 7f1688d3a640 1 --2- 172.21.15.7:0/1481036120 >> v2:172.21.15.177:3300/0 conn(0x7f168406f670 0x7f16840717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.443+0000 7f1688d3a640 1 -- 172.21.15.7:0/1481036120 >> v1:172.21.15.177:6789/0 conn(0x7f1684146200 legacy=0x7f1684146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.443+0000 7f1688d3a640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.443+0000 7f1688d3a640 1 monclient: found mon.noname-a 2024-01-21T03:20:04.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.443+0000 7f1688d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.443+0000 7f1688d3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.443+0000 7f1688d3a640 1 -- 172.21.15.7:0/1481036120 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16840adb40 con 0x7f168406f290 2024-01-21T03:20:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.445+0000 7fabf77fe640 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2024-01-21T03:20:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.445+0000 7fabf77fe640 1 -- 172.21.15.7:0/4292718646 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabf00326d0 msgr2=0x7fabf00349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.445+0000 7fabf77fe640 1 --2- 172.21.15.7:0/4292718646 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabf00326d0 0x7fabf00349b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.445+0000 7fabf77fe640 1 -- 172.21.15.7:0/4292718646 >> v2:172.21.15.7:3300/0 conn(0x7fabf0035620 msgr2=0x7fabf00c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.445+0000 7fabf77fe640 1 --2- 172.21.15.7:0/4292718646 >> v2:172.21.15.7:3300/0 conn(0x7fabf0035620 0x7fabf00c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.445+0000 7fabf77fe640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.445+0000 7fabf77fe640 1 monclient: found mon.noname-c 2024-01-21T03:20:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.445+0000 7fabf77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.445+0000 7fabf77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.445+0000 7fabf77fe640 1 -- 172.21.15.7:0/4292718646 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf0029590 con 0x7fabf0035240 2024-01-21T03:20:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.448+0000 7fabf77fe640 1 --2- 172.21.15.7:0/4292718646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf0035240 0x7fabf002f3f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fabf000aa40 tx=0x7fabec02b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38c88719071995c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.449+0000 7f1688d3a640 1 --2- 172.21.15.7:0/1481036120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f168406f290 0x7f16840712b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16841472c0 tx=0x7f167c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bd90318b95a9be server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.451+0000 7ff23d061640 10 monclient: get_monmap_and_config 2024-01-21T03:20:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.451+0000 7ff23d061640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.451+0000 7ff23d061640 10 monclient: monmap: 2024-01-21T03:20:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.452908+0000 2024-01-21T03:20:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.452908+0000 2024-01-21T03:20:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:04.438 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.439 INFO:teuthology.orchestra.run.smithi007.stdout:60129542152 2024-01-21T03:20:04.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-21T03:20:04.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.452+0000 7ff23d061640 1 Processor -- start 2024-01-21T03:20:04.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7ff23d061640 1 -- start start 2024-01-21T03:20:04.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7ff23d061640 10 monclient: init 2024-01-21T03:20:04.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7ff23d061640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7ff23d061640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-01-21T03:20:04.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7ff23d061640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff230038db0 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:04.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7ff23d061640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff230039190 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:04.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7ff23d061640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2300c8a60 0x7ff2300d3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7ff23d061640 10 monclient(hunting): picked mon.noname-b con 0x7ff2300c8a60 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7ff23d061640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2300c7ea0 con 0x7ff230039190 2024-01-21T03:20:04.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7ff23d061640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff230035470 con 0x7ff230038db0 2024-01-21T03:20:04.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7ff23d061640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7ff23d061640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2300c5970 con 0x7ff2300c8a60 2024-01-21T03:20:04.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7ff23d061640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7ff23d061640 10 monclient(hunting): authenticate will time out at 1350.769918s 2024-01-21T03:20:04.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabf67fc640 1 -- 172.21.15.7:0/4292718646 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec031750 con 0x7fabf0035240 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabf67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabf67fc640 10 monclient: dump: 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabf67fc640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabf67fc640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabf67fc640 1 -- 172.21.15.7:0/4292718646 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabec0318b0 con 0x7fabf0035240 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabf67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabf67fc640 1 -- 172.21.15.7:0/4292718646 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec031c80 con 0x7fabf0035240 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabf67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabf67fc640 10 monclient: dump: 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.446 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabfc947640 5 monclient: authenticate success, global_id 4321 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabfc947640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.453+0000 7fabfc947640 4 set_mon_vals no callback set 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7fabfc947640 10 monclient: shutdown 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7fabfc947640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7fabfc947640 1 -- 172.21.15.7:0/4292718646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf0035240 msgr2=0x7fabf002f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7fabfc947640 1 --2- 172.21.15.7:0/4292718646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf0035240 0x7fabf002f3f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fabf000aa40 tx=0x7fabec02b7f0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7fabfc947640 1 -- 172.21.15.7:0/4292718646 shutdown_connections 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7fabfc947640 1 --2- 172.21.15.7:0/4292718646 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabf00326d0 0x7fabf00349b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7fabfc947640 1 --2- 172.21.15.7:0/4292718646 >> v2:172.21.15.7:3300/0 conn(0x7fabf0035620 0x7fabf00c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7fabfc947640 1 --2- 172.21.15.7:0/4292718646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf0035240 0x7fabf002f3f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.454+0000 7fabfc947640 1 -- 172.21.15.7:0/4292718646 >> 172.21.15.7:0/4292718646 conn(0x7fabf00060c0 msgr2=0x7fabf00064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.455+0000 7fabfc947640 1 -- 172.21.15.7:0/4292718646 shutdown_connections 2024-01-21T03:20:04.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.455+0000 7fabfc947640 1 -- 172.21.15.7:0/4292718646 wait complete. 2024-01-21T03:20:04.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.456+0000 7ff236d9d640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7ff230038db0 0x7ff2300c5260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:44384/0 (socket says 172.21.15.7:44384) 2024-01-21T03:20:04.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.456+0000 7ff236d9d640 1 -- 172.21.15.7:0/397666239 learned_addr learned my addr 172.21.15.7:0/397666239 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:04.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.457+0000 7fabfc947640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.457+0000 7fabfc947640 10 monclient: monmap: 2024-01-21T03:20:04.448 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.448 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.449 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.449 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.449 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.449 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.449 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.449 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.449 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.449 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.457+0000 7fabfc947640 1 librados: starting msgr at 2024-01-21T03:20:04.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.457+0000 7fabfc947640 1 librados: starting objecter 2024-01-21T03:20:04.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.457+0000 7fabfc947640 1 Processor -- start 2024-01-21T03:20:04.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.458+0000 7fabfc947640 1 -- start start 2024-01-21T03:20:04.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.458+0000 7fabfc947640 1 librados: setting wanted keys 2024-01-21T03:20:04.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.458+0000 7fabfc947640 1 librados: calling monclient init 2024-01-21T03:20:04.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.458+0000 7fabfc947640 10 monclient: init 2024-01-21T03:20:04.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.459+0000 7fabfc947640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.459+0000 7fabfc947640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:20:04.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.459+0000 7fabfc947640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf00326d0 0x7fabf00c47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.459+0000 7fabfc947640 10 monclient(hunting): picked mon.noname-a con 0x7fabf00326d0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.459+0000 7fabfc947640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf0035240 0x7fabf00b8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.459+0000 7fabfc947640 10 monclient(hunting): picked mon.noname-b con 0x7fabf0035240 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.460+0000 7fabfc947640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabf00bd9f0 0x7fabf00c1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.460+0000 7fabfc947640 10 monclient(hunting): picked mon.noname-c con 0x7fabf00bd9f0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.460+0000 7fabfc947640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.460+0000 7fabfc947640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf002efa0 con 0x7fabf00326d0 2024-01-21T03:20:04.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.460+0000 7fabfc947640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.460+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750115968 0 0) 0x7ff230035470 con 0x7ff230038db0 2024-01-21T03:20:04.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.460+0000 7ff235d9b640 10 monclient(hunting): _negotiate 2024-01-21T03:20:04.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.460+0000 7ff235d9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.460+0000 7ff235d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.460+0000 7ff235d9b640 10 monclient(hunting): my global_id is 4327 2024-01-21T03:20:04.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.460+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff220003520 con 0x7ff230038db0 2024-01-21T03:20:04.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.460+0000 7fabf6ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf0035240 0x7fabf00b8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.461+0000 7fabf6ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf0035240 0x7fabf00b8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36684/0 (socket says 172.21.15.7:36684) 2024-01-21T03:20:04.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.461+0000 7fabf6ffd640 1 -- 172.21.15.7:0/1199614872 learned_addr learned my addr 172.21.15.7:0/1199614872 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.460+0000 7fabfc947640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00bace0 con 0x7fabf0035240 2024-01-21T03:20:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.461+0000 7fabfc947640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.461+0000 7fabfc947640 1 -- 172.21.15.7:0/1199614872 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00c4d20 con 0x7fabf00bd9f0 2024-01-21T03:20:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.461+0000 7fabfc947640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.461+0000 7fabfc947640 10 monclient(hunting): authenticate will time out at 1350.777194s 2024-01-21T03:20:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.461+0000 7fabf6ffd640 10 monclient(hunting): get_auth_request con 0x7fabf0035240 auth_method 0 2024-01-21T03:20:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.461+0000 7fabf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.461+0000 7fabf6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.461+0000 7fabf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.463+0000 7fabf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.463+0000 7fabf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.464+0000 7fabf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.464+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3312417321 0 0) 0x7ff220003520 con 0x7ff230038db0 2024-01-21T03:20:04.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.464+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2300c8a60 msgr2=0x7ff2300d3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.464+0000 7ff235d9b640 1 --2- 172.21.15.7:0/397666239 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2300c8a60 0x7ff2300d3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.464+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 >> v1:172.21.15.7:6789/0 conn(0x7ff230039190 legacy=0x7ff2300c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 1 monclient: found mon.noname-c-legacy 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.177:6789/0 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2300358a0 con 0x7ff230038db0 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 749591385 0 0) 0x7ff228001670 con 0x7ff230038db0 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient: dump: 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] but i'm connected to v1:172.21.15.177:6789/0, reconnecting 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 >> v1:172.21.15.177:6789/0 conn(0x7ff230038db0 legacy=0x7ff2300c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 1 --2- 172.21.15.7:0/397666239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff220003970 0x7ff220003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient(hunting): picked mon.a con 0x7ff220003970 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 1 --2- 172.21.15.7:0/397666239 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2200047c0 0x7ff220004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient(hunting): picked mon.c con 0x7ff2200047c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 1 --2- 172.21.15.7:0/397666239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2200052a0 0x7ff220014190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient(hunting): picked mon.b con 0x7ff2200052a0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2300c5970 con 0x7ff220003970 2024-01-21T03:20:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff220014980 con 0x7ff2200052a0 2024-01-21T03:20:04.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff220014b40 con 0x7ff2200047c0 2024-01-21T03:20:04.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7ff235d9b640 10 monclient(hunting): _finish_auth 0 2024-01-21T03:20:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 1 -- 172.21.15.7:0/3891581075 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f00f0058f70 con 0x7f00f8142b80 2024-01-21T03:20:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 10 client.4287.objecter ms_dispatch 0x7f00f8000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 3 client.4287.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 3 client.4287.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 10 client.4287.objecter checking command 1 2024-01-21T03:20:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 1 --2- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] conn(0x7f00dc072c90 0x7f00dc0750b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 20 client.4287.objecter _get_session s=0x7f00dc0725c0 osd=2 3 2024-01-21T03:20:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 20 client.4287.objecter put_session s=0x7f00dc0725c0 osd=2 3 2024-01-21T03:20:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 15 client.4287.objecter _session_command_op_remove -1 1 2024-01-21T03:20:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 20 client.4287.objecter _get_session s=0x7f00dc0725c0 osd=2 3 2024-01-21T03:20:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 20 client.4287.objecter get_session s=0x7f00dc0725c0 osd=2 3 2024-01-21T03:20:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 15 client.4287.objecter _session_command_op_assign 2 1 2024-01-21T03:20:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 20 client.4287.objecter put_session s=0x7f00dc0725c0 osd=2 4 2024-01-21T03:20:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 10 client.4287.objecter _send_command 1 2024-01-21T03:20:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 1 -- 172.21.15.7:0/3891581075 --> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00dc075760 con 0x7f00dc072c90 2024-01-21T03:20:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 20 client.4287.objecter dump_active .. 0 homeless 2024-01-21T03:20:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 1 -- 172.21.15.7:0/3891581075 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00f00098d0 con 0x7f00f8142b80 2024-01-21T03:20:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.465+0000 7f00edffb640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-21T03:20:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.466+0000 7f00fcd7d640 1 --2- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] conn(0x7f00dc072c90 0x7f00dc0750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.467+0000 7f00fcd7d640 10 monclient: get_auth_request con 0x7f00dc072c90 auth_method 0 2024-01-21T03:20:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.467+0000 7fabf6ffd640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2024-01-21T03:20:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.467+0000 7fabf6ffd640 1 -- 172.21.15.7:0/1199614872 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabf00bd9f0 msgr2=0x7fabf00c1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.467+0000 7fabf6ffd640 1 --2- 172.21.15.7:0/1199614872 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabf00bd9f0 0x7fabf00c1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.467+0000 7fabf6ffd640 1 -- 172.21.15.7:0/1199614872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf00326d0 msgr2=0x7fabf00c47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.467+0000 7fabf6ffd640 1 --2- 172.21.15.7:0/1199614872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf00326d0 0x7fabf00c47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.467+0000 7fabf6ffd640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.467+0000 7fabf6ffd640 1 monclient: found mon.noname-b 2024-01-21T03:20:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.467+0000 7fabf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.467+0000 7fabf6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.467+0000 7fabf6ffd640 1 -- 172.21.15.7:0/1199614872 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabec009400 con 0x7fabf0035240 2024-01-21T03:20:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.469+0000 7ff23659c640 1 --2- 172.21.15.7:0/397666239 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2200047c0 0x7ff220004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.471+0000 7ff23759e640 1 --2- 172.21.15.7:0/397666239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2200052a0 0x7ff220014190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.471+0000 7fabf6ffd640 1 --2- 172.21.15.7:0/1199614872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf0035240 0x7fabf00b8fe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fabe400e7e0 tx=0x7fabe400eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.471+0000 7fabe3fff640 10 client.?.objecter ms_handle_connect 0x7fabf0035240 2024-01-21T03:20:04.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.471+0000 7fabe3fff640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.471+0000 7ff23759e640 10 monclient(hunting): get_auth_request con 0x7ff2200052a0 auth_method 0 2024-01-21T03:20:04.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.471+0000 7ff23759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.471+0000 7ff23759e640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.471+0000 7ff23759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:20:04.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.474+0000 7f00fcd7d640 1 --2- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] conn(0x7f00dc072c90 0x7f00dc0750b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.474+0000 7f00edffb640 10 client.4287.objecter ms_handle_connect 0x7f00dc072c90 2024-01-21T03:20:04.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.472+0000 7f16837fe640 1 -- 172.21.15.7:0/1481036120 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c02ca80 con 0x7f168406f290 2024-01-21T03:20:04.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.472+0000 7f16837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.472+0000 7f16837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:04.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.472+0000 7f16837fe640 10 monclient: dump: 2024-01-21T03:20:04.465 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.472+0000 7f16837fe640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.472+0000 7f16837fe640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.472+0000 7f16837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.473+0000 7f1689d3c640 5 monclient: authenticate success, global_id 4293 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.473+0000 7f1689d3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:04.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.473+0000 7f16837fe640 1 -- 172.21.15.7:0/1481036120 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167c02cbe0 con 0x7f168406f290 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.473+0000 7f16837fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.473+0000 7f1689d3c640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.473+0000 7f1689d3c640 4 set_mon_vals no callback set 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.473+0000 7f1689d3c640 10 monclient: shutdown 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.473+0000 7f1689d3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.473+0000 7f1689d3c640 1 -- 172.21.15.7:0/1481036120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f168406f290 msgr2=0x7f16840712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.473+0000 7f1689d3c640 1 --2- 172.21.15.7:0/1481036120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f168406f290 0x7f16840712b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16841472c0 tx=0x7f167c02bee0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.474+0000 7f1689d3c640 1 -- 172.21.15.7:0/1481036120 shutdown_connections 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.474+0000 7f1689d3c640 1 --2- 172.21.15.7:0/1481036120 >> v2:172.21.15.177:3300/0 conn(0x7f168406f670 0x7f16840717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.474+0000 7f1689d3c640 1 --2- 172.21.15.7:0/1481036120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f168406f290 0x7f16840712b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.474+0000 7f1689d3c640 1 -- 172.21.15.7:0/1481036120 >> 172.21.15.7:0/1481036120 conn(0x7f16840b0040 msgr2=0x7f16840b0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.474+0000 7f1689d3c640 1 -- 172.21.15.7:0/1481036120 shutdown_connections 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.474+0000 7f1689d3c640 1 -- 172.21.15.7:0/1481036120 wait complete. 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.475+0000 7f1689d3c640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.475+0000 7f1689d3c640 10 monclient: monmap: 2024-01-21T03:20:04.467 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.468 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.475+0000 7ff23759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.475+0000 7ff23759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.475+0000 7ff23759e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T03:20:04.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabe3fff640 1 -- 172.21.15.7:0/1199614872 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe4004030 con 0x7fabf0035240 2024-01-21T03:20:04.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabe3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:04.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabe3fff640 10 monclient: dump: 2024-01-21T03:20:04.469 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.469 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.469 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.469 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabe3fff640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabe3fff640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabe3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabfc947640 5 monclient: authenticate success, global_id 4330 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabfc947640 10 monclient: _renew_subs 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabfc947640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabfc947640 1 -- 172.21.15.7:0/1199614872 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf002a980 con 0x7fabf0035240 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabfc947640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabfc947640 10 monclient: _renew_subs 2024-01-21T03:20:04.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabfc947640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabfc947640 1 -- 172.21.15.7:0/1199614872 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf00b9520 con 0x7fabf0035240 2024-01-21T03:20:04.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fabfc947640 1 librados: init done 2024-01-21T03:20:04.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7ff236d9d640 1 --2- 172.21.15.7:0/397666239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff220003970 0x7ff220003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fb397a0e640 10 monclient: get_monmap_and_config 2024-01-21T03:20:04.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fb397a0e640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.476+0000 7fb397a0e640 10 monclient: monmap: 2024-01-21T03:20:04.471 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.471 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.472 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.477568+0000 2024-01-21T03:20:04.472 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.477568+0000 2024-01-21T03:20:04.472 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.472 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.472 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:04.472 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:04.472 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:04.472 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:04.472 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:04.472 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.475+0000 7f1689d3c640 1 librados: starting msgr at 2024-01-21T03:20:04.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.475+0000 7f1689d3c640 1 librados: starting objecter 2024-01-21T03:20:04.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.475+0000 7f1689d3c640 1 Processor -- start 2024-01-21T03:20:04.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.475+0000 7f1689d3c640 1 -- start start 2024-01-21T03:20:04.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.475+0000 7f1689d3c640 1 librados: setting wanted keys 2024-01-21T03:20:04.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.475+0000 7f1689d3c640 1 librados: calling monclient init 2024-01-21T03:20:04.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.475+0000 7f1689d3c640 10 monclient: init 2024-01-21T03:20:04.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7fabe3fff640 1 -- 172.21.15.7:0/1199614872 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe4004190 con 0x7fabf0035240 2024-01-21T03:20:04.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.477+0000 7ff236d9d640 10 monclient(hunting): get_auth_request con 0x7ff220003970 auth_method 0 2024-01-21T03:20:04.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.477+0000 7ff236d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.477+0000 7ff236d9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.477+0000 7ff236d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23759e640 10 monclient(hunting): handle_auth_done global_id 4327 payload 293 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23759e640 1 -- 172.21.15.7:0/397666239 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2200047c0 msgr2=0x7ff220004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23759e640 1 --2- 172.21.15.7:0/397666239 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2200047c0 0x7ff220004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23759e640 1 -- 172.21.15.7:0/397666239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff220003970 msgr2=0x7ff220003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23759e640 1 --2- 172.21.15.7:0/397666239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff220003970 0x7ff220003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23759e640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23759e640 1 monclient: found mon.b 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23759e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23759e640 1 -- 172.21.15.7:0/397666239 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2300358a0 con 0x7ff2200052a0 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23759e640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23759e640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23759e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23d061640 5 monclient: authenticate success, global_id 4327 2024-01-21T03:20:04.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7ff23d061640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:04.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7fabe3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T03:20:04.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f168406f290 0x7f16841459b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 10 monclient(hunting): picked mon.noname-c con 0x7f168406f290 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f168406f670 0x7f16840abf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 10 monclient(hunting): picked mon.noname-a con 0x7f168406f670 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1684146200 0x7f16840ac450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 10 monclient(hunting): picked mon.noname-b con 0x7f1684146200 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16840ab360 con 0x7f168406f670 2024-01-21T03:20:04.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16840a8e90 con 0x7f1684146200 2024-01-21T03:20:04.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16840ab560 con 0x7f168406f290 2024-01-21T03:20:04.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7f1689d3c640 10 monclient(hunting): authenticate will time out at 1350.794616s 2024-01-21T03:20:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7fabf5ffb640 4 set_mon_vals no callback set 2024-01-21T03:20:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.478+0000 7fabe3fff640 1 -- 172.21.15.7:0/1199614872 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe4004520 con 0x7fabf0035240 2024-01-21T03:20:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.479+0000 7fabe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.479+0000 7fabe3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.479+0000 7fabe3fff640 10 monclient: dump: 2024-01-21T03:20:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.478 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.479+0000 7fabe1ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.479+0000 7fb397a0e640 1 Processor -- start 2024-01-21T03:20:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.479+0000 7fabe1ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.479+0000 7fabe1ffb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.479+0000 7fabe1ffb640 1 -- 172.21.15.7:0/1199614872 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd8005080 con 0x7fabf0035240 2024-01-21T03:20:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.479+0000 7fb397a0e640 1 -- start start 2024-01-21T03:20:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.479+0000 7fb397a0e640 10 monclient: init 2024-01-21T03:20:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7f1688d3a640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f168406f290 0x7f16841459b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7f1688d3a640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f168406f290 0x7f16841459b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45946/0 (socket says 172.21.15.7:45946) 2024-01-21T03:20:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7f1688d3a640 1 -- 172.21.15.7:0/2317196430 learned_addr learned my addr 172.21.15.7:0/2317196430 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7f1688d3a640 10 monclient(hunting): get_auth_request con 0x7f168406f290 auth_method 0 2024-01-21T03:20:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7f1688d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7f1688d3a640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7f1688d3a640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-01-21T03:20:04.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 0x7fb39013b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 10 monclient(hunting): picked mon.noname-b con 0x7fb3900b3e40 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb39006e990 0x7fb390071550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 10 monclient(hunting): picked mon.noname-c con 0x7fb39006e990 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:04.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3901417e0 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:04.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb39006f170 con 0x7fb3901417e0 2024-01-21T03:20:04.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb390070160 con 0x7fb39006e990 2024-01-21T03:20:04.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb390142340 con 0x7fb3900b3e40 2024-01-21T03:20:04.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.480+0000 7fb397a0e640 10 monclient(hunting): authenticate will time out at 1350.796679s 2024-01-21T03:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.482+0000 7f168953b640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1684146200 0x7f16840ac450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.483+0000 7f1688d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.483+0000 7f1688d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.483+0000 7f1688d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.484+0000 7f1683fff640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f168406f670 0x7f16840abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.485+0000 7f1683fff640 10 monclient(hunting): get_auth_request con 0x7f168406f670 auth_method 0 2024-01-21T03:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.485+0000 7f1683fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.485+0000 7f1683fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.485+0000 7f1683fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.485+0000 7f1683fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.485+0000 7f1683fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.485+0000 7f1683fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.488+0000 7f1683fff640 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.488+0000 7f1683fff640 1 -- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f168406f290 msgr2=0x7f16841459b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.488+0000 7f1683fff640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f168406f290 0x7f16841459b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.488+0000 7f1683fff640 1 -- 172.21.15.7:0/2317196430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1684146200 msgr2=0x7f16840ac450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.488+0000 7f1683fff640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1684146200 0x7f16840ac450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.488+0000 7f1683fff640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.488+0000 7f1683fff640 1 monclient: found mon.noname-a 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.488+0000 7f1683fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.488+0000 7f1683fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.488+0000 7f1683fff640 1 -- 172.21.15.7:0/2317196430 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c009b30 con 0x7f168406f670 2024-01-21T03:20:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.491+0000 7f1683fff640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f168406f670 0x7f16840abf10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f167400e850 tx=0x7f167400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.491+0000 7f16817fa640 10 client.?.objecter ms_handle_connect 0x7f168406f670 2024-01-21T03:20:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.491+0000 7f16817fa640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.496+0000 7fb3957aa640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 0x7fb39013b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7fb3957aa640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 0x7fb39013b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45952/0 (socket says 172.21.15.7:45952) 2024-01-21T03:20:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7fb3957aa640 1 -- 172.21.15.7:0/1592107879 learned_addr learned my addr 172.21.15.7:0/1592107879 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7fb3957aa640 10 monclient(hunting): get_auth_request con 0x7fb3900b3e40 auth_method 0 2024-01-21T03:20:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7fb3957aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7fb3957aa640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7fb3957aa640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.499+0000 7fb3957aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.499+0000 7fb3957aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.499+0000 7fb3957aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f16817fa640 1 -- 172.21.15.7:0/2317196430 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16740095b0 con 0x7f168406f670 2024-01-21T03:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f16817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f16817fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f16817fa640 10 monclient: dump: 2024-01-21T03:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f16817fa640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f16817fa640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f16817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f1689d3c640 5 monclient: authenticate success, global_id 4299 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f1689d3c640 10 monclient: _renew_subs 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f1689d3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f1689d3c640 1 -- 172.21.15.7:0/2317196430 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16840ac990 con 0x7f168406f670 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f1689d3c640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f1689d3c640 10 monclient: _renew_subs 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f1689d3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f1689d3c640 1 -- 172.21.15.7:0/2317196430 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16841327c0 con 0x7f168406f670 2024-01-21T03:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f1689d3c640 1 librados: init done 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f16817fa640 1 -- 172.21.15.7:0/2317196430 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167400c940 con 0x7f168406f670 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f16817fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.498+0000 7f16827fc640 4 set_mon_vals no callback set 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.499+0000 7f16817fa640 1 -- 172.21.15.7:0/2317196430 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167400ccd0 con 0x7f168406f670 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.499+0000 7f16817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.499+0000 7f16817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.499+0000 7f16817fa640 10 monclient: dump: 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.499+0000 7f1689d3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.499+0000 7f1689d3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.499+0000 7f1689d3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.499+0000 7f1689d3c640 1 -- 172.21.15.7:0/2317196430 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16840abb60 con 0x7f168406f670 2024-01-21T03:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.503+0000 7fb3957aa640 10 monclient(hunting): handle_auth_done global_id 4286 payload 274 2024-01-21T03:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.503+0000 7fb3957aa640 1 -- 172.21.15.7:0/1592107879 >> v2:172.21.15.177:3300/0 conn(0x7fb39006e990 msgr2=0x7fb390071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.503+0000 7fb3957aa640 1 --2- 172.21.15.7:0/1592107879 >> v2:172.21.15.177:3300/0 conn(0x7fb39006e990 0x7fb390071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.504+0000 7fb3957aa640 1 -- 172.21.15.7:0/1592107879 >> v1:172.21.15.177:6789/0 conn(0x7fb3901417e0 legacy=0x7fb390141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.504+0000 7fb3957aa640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.504+0000 7fb3957aa640 1 monclient: found mon.noname-b 2024-01-21T03:20:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.504+0000 7fb3957aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.504+0000 7fb3957aa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.504+0000 7fb3957aa640 1 -- 172.21.15.7:0/1592107879 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb390070ec0 con 0x7fb3900b3e40 2024-01-21T03:20:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.504+0000 7fe2d74c2640 10 monclient: get_monmap_and_config 2024-01-21T03:20:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.504+0000 7fe2d74c2640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.504+0000 7fe2d74c2640 10 monclient: monmap: 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.505760+0000 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.505760+0000 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.505+0000 7fe2d74c2640 1 Processor -- start 2024-01-21T03:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.505+0000 7fe2d74c2640 1 -- start start 2024-01-21T03:20:04.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.505+0000 7fe2d74c2640 10 monclient: init 2024-01-21T03:20:04.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.506+0000 7fe2d74c2640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.506+0000 7fe2d74c2640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-01-21T03:20:04.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.506+0000 7fb3957aa640 1 --2- 172.21.15.7:0/1592107879 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 0x7fb39013b470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb388009e50 tx=0x7fb38802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a8e2cfeae203e13 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.506+0000 7fe2d74c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2d00a8b20 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.507+0000 7fe2d74c2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2d00a9d10 0x7fe2d013f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.507+0000 7fe2d74c2640 10 monclient(hunting): picked mon.noname-a con 0x7fe2d00a9d10 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.507+0000 7fe2d74c2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00b2310 0x7fe2d013f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.507+0000 7fe2d74c2640 10 monclient(hunting): picked mon.noname-b con 0x7fe2d00b2310 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.507+0000 7fe2d74c2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d01456f0 con 0x7fe2d00a8b20 2024-01-21T03:20:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.507+0000 7fe2d74c2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.507+0000 7fe2d74c2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2d00a98c0 con 0x7fe2d00a9d10 2024-01-21T03:20:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.507+0000 7fe2d74c2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.507+0000 7fe2d74c2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d013fb00 con 0x7fe2d00b2310 2024-01-21T03:20:04.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.507+0000 7fe2d74c2640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.507+0000 7fe2d74c2640 10 monclient(hunting): authenticate will time out at 1350.823010s 2024-01-21T03:20:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.512+0000 7fe2d5a5f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00b2310 0x7fe2d013f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.513+0000 7fb387fff640 1 -- 172.21.15.7:0/1592107879 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38802c9e0 con 0x7fb3900b3e40 2024-01-21T03:20:04.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.513+0000 7fb387fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.513+0000 7fb387fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.513+0000 7fb387fff640 10 monclient: dump: 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.513+0000 7fb387fff640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.513+0000 7fb387fff640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.513+0000 7fb387fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.513+0000 7fb397a0e640 5 monclient: authenticate success, global_id 4286 2024-01-21T03:20:04.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.513+0000 7fb397a0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:04.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.513+0000 7fb387fff640 1 -- 172.21.15.7:0/1592107879 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38802cb40 con 0x7fb3900b3e40 2024-01-21T03:20:04.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.513+0000 7fb387fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.513+0000 7fb397a0e640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.513+0000 7fb397a0e640 4 set_mon_vals no callback set 2024-01-21T03:20:04.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fe2d5a5f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00b2310 0x7fe2d013f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45964/0 (socket says 172.21.15.7:45964) 2024-01-21T03:20:04.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fe2d5a5f640 1 -- 172.21.15.7:0/1035376643 learned_addr learned my addr 172.21.15.7:0/1035376643 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fe2d5a5f640 10 monclient(hunting): get_auth_request con 0x7fe2d00b2310 auth_method 0 2024-01-21T03:20:04.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fe2d5a5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fe2d5a5f640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fe2d5a5f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fb387fff640 1 -- 172.21.15.7:0/1592107879 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3880343f0 con 0x7fb3900b3e40 2024-01-21T03:20:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fb387fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fb387fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fb387fff640 10 monclient: dump: 2024-01-21T03:20:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.504 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fb397a0e640 10 monclient: shutdown 2024-01-21T03:20:04.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fb397a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fb397a0e640 1 -- 172.21.15.7:0/1592107879 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 msgr2=0x7fb39013b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fb397a0e640 1 --2- 172.21.15.7:0/1592107879 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 0x7fb39013b470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb388009e50 tx=0x7fb38802be40 comp rx=0 tx=0).stop 2024-01-21T03:20:04.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fb397a0e640 1 -- 172.21.15.7:0/1592107879 shutdown_connections 2024-01-21T03:20:04.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fb397a0e640 1 --2- 172.21.15.7:0/1592107879 >> v2:172.21.15.177:3300/0 conn(0x7fb39006e990 0x7fb390071550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.514+0000 7fb397a0e640 1 --2- 172.21.15.7:0/1592107879 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 0x7fb39013b470 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.515+0000 7fb397a0e640 1 -- 172.21.15.7:0/1592107879 >> 172.21.15.7:0/1592107879 conn(0x7fb39006b270 msgr2=0x7fb3900b3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.515+0000 7fb397a0e640 1 -- 172.21.15.7:0/1592107879 shutdown_connections 2024-01-21T03:20:04.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.515+0000 7fb397a0e640 1 -- 172.21.15.7:0/1592107879 wait complete. 2024-01-21T03:20:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.515+0000 7fe2d5a5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.515+0000 7fe2d5a5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.515+0000 7fe2d5a5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.516+0000 7fb397a0e640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.516+0000 7fb397a0e640 10 monclient: monmap: 2024-01-21T03:20:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.516+0000 7fb397a0e640 1 librados: starting msgr at 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.516+0000 7fb397a0e640 1 librados: starting objecter 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.516+0000 7fb397a0e640 1 Processor -- start 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.516+0000 7fb397a0e640 1 -- start start 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.516+0000 7fb397a0e640 1 librados: setting wanted keys 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.516+0000 7fb397a0e640 1 librados: calling monclient init 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.516+0000 7fb397a0e640 10 monclient: init 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb39006e990 0x7fb39013daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 10 monclient(hunting): picked mon.noname-b con 0x7fb39006e990 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 0x7fb390136f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 10 monclient(hunting): picked mon.noname-c con 0x7fb3900b3e40 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3901417e0 0x7fb3901374c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 10 monclient(hunting): picked mon.noname-a con 0x7fb3901417e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3901323c0 con 0x7fb3901417e0 2024-01-21T03:20:04.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb390138360 con 0x7fb39006e990 2024-01-21T03:20:04.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39013a250 con 0x7fb3900b3e40 2024-01-21T03:20:04.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.517+0000 7fb397a0e640 10 monclient(hunting): authenticate will time out at 1350.833534s 2024-01-21T03:20:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.518+0000 7fe2d5a5f640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2024-01-21T03:20:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.518+0000 7fe2d5a5f640 1 -- 172.21.15.7:0/1035376643 >> v2:172.21.15.7:3300/0 conn(0x7fe2d00a9d10 msgr2=0x7fe2d013f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.518+0000 7fe2d5a5f640 1 --2- 172.21.15.7:0/1035376643 >> v2:172.21.15.7:3300/0 conn(0x7fe2d00a9d10 0x7fe2d013f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.518+0000 7fe2d5a5f640 1 -- 172.21.15.7:0/1035376643 >> v1:172.21.15.177:6789/0 conn(0x7fe2d00a8b20 legacy=0x7fe2d0145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.518+0000 7fe2d5a5f640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.518+0000 7fe2d5a5f640 1 monclient: found mon.noname-b 2024-01-21T03:20:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.518+0000 7fe2d5a5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.518+0000 7fe2d5a5f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.518+0000 7fe2d5a5f640 1 -- 172.21.15.7:0/1035376643 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d0146290 con 0x7fe2d00b2310 2024-01-21T03:20:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.523+0000 7fe2d5a5f640 1 --2- 172.21.15.7:0/1035376643 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00b2310 0x7fe2d013f5c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8009e50 tx=0x7fe2c802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1d7f58ccb1799e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.524+0000 7fb394fa9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 0x7fb390136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.527+0000 7fb394fa9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 0x7fb390136f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45972/0 (socket says 172.21.15.7:45972) 2024-01-21T03:20:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.527+0000 7fb394fa9640 1 -- 172.21.15.7:0/124933700 learned_addr learned my addr 172.21.15.7:0/124933700 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.527+0000 7fb394fa9640 10 monclient(hunting): get_auth_request con 0x7fb3900b3e40 auth_method 0 2024-01-21T03:20:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.527+0000 7fb394fa9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.527+0000 7fb394fa9640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.527+0000 7fb394fa9640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.530+0000 7fb394fa9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.530+0000 7fb394fa9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.530+0000 7fb394fa9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2c7fff640 1 -- 172.21.15.7:0/1035376643 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c802ca70 con 0x7fe2d00b2310 2024-01-21T03:20:04.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:04.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2c7fff640 10 monclient: dump: 2024-01-21T03:20:04.517 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.517 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.517 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.517 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.517 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.517 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.517 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.517 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.517 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2c7fff640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2c7fff640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2c7fff640 1 -- 172.21.15.7:0/1035376643 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c802cbd0 con 0x7fe2d00b2310 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2c7fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2c7fff640 1 -- 172.21.15.7:0/1035376643 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c80343f0 con 0x7fe2d00b2310 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2c7fff640 10 monclient: dump: 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.518 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.519 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.519 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.519 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2d74c2640 5 monclient: authenticate success, global_id 4289 2024-01-21T03:20:04.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2d74c2640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.531+0000 7fe2d74c2640 4 set_mon_vals no callback set 2024-01-21T03:20:04.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.532+0000 7fe2d74c2640 10 monclient: shutdown 2024-01-21T03:20:04.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.532+0000 7fe2d74c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.532+0000 7fe2d74c2640 1 -- 172.21.15.7:0/1035376643 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00b2310 msgr2=0x7fe2d013f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.532+0000 7fe2d74c2640 1 --2- 172.21.15.7:0/1035376643 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00b2310 0x7fe2d013f5c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8009e50 tx=0x7fe2c802bed0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.532+0000 7fe2d74c2640 1 -- 172.21.15.7:0/1035376643 shutdown_connections 2024-01-21T03:20:04.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.532+0000 7fe2d74c2640 1 --2- 172.21.15.7:0/1035376643 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00b2310 0x7fe2d013f5c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.532+0000 7fe2d74c2640 1 --2- 172.21.15.7:0/1035376643 >> v2:172.21.15.7:3300/0 conn(0x7fe2d00a9d10 0x7fe2d013f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.532+0000 7fe2d74c2640 1 -- 172.21.15.7:0/1035376643 >> 172.21.15.7:0/1035376643 conn(0x7fe2d0066fb0 msgr2=0x7fe2d00b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.532+0000 7fe2d74c2640 1 -- 172.21.15.7:0/1035376643 shutdown_connections 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.532+0000 7fe2d74c2640 1 -- 172.21.15.7:0/1035376643 wait complete. 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.533+0000 7fe2d74c2640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.533+0000 7fe2d74c2640 10 monclient: monmap: 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.534+0000 7fe2d74c2640 1 librados: starting msgr at 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.534+0000 7fe2d74c2640 1 librados: starting objecter 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.534+0000 7fe2d74c2640 1 Processor -- start 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.534+0000 7fe2d74c2640 1 -- start start 2024-01-21T03:20:04.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.534+0000 7fe2d74c2640 1 librados: setting wanted keys 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.534+0000 7fe2d74c2640 1 librados: calling monclient init 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.534+0000 7fe2d74c2640 10 monclient: init 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00a8b20 0x7fe2d0133e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 10 monclient(hunting): picked mon.noname-c con 0x7fe2d00a8b20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2d00a9d10 0x7fe2d0133460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 10 monclient(hunting): picked mon.noname-b con 0x7fe2d00a9d10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2d00b2310 0x7fe2d01d7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 10 monclient(hunting): picked mon.noname-a con 0x7fe2d00b2310 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d0134360 con 0x7fe2d00b2310 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d01344d0 con 0x7fe2d00a9d10 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d0143800 con 0x7fe2d00a8b20 2024-01-21T03:20:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.535+0000 7fe2d74c2640 10 monclient(hunting): authenticate will time out at 1350.851233s 2024-01-21T03:20:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.536+0000 7fb394fa9640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-01-21T03:20:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.536+0000 7fb394fa9640 1 -- 172.21.15.7:0/124933700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb39006e990 msgr2=0x7fb39013daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.536+0000 7fb394fa9640 1 --2- 172.21.15.7:0/124933700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb39006e990 0x7fb39013daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.536+0000 7fb394fa9640 1 -- 172.21.15.7:0/124933700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3901417e0 msgr2=0x7fb3901374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.536+0000 7fb394fa9640 1 --2- 172.21.15.7:0/124933700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3901417e0 0x7fb3901374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.536+0000 7fb394fa9640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.536+0000 7fb394fa9640 1 monclient: found mon.noname-c 2024-01-21T03:20:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.536+0000 7fb394fa9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.536+0000 7fb394fa9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.536+0000 7fb394fa9640 1 -- 172.21.15.7:0/124933700 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb388009b30 con 0x7fb3900b3e40 2024-01-21T03:20:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.540+0000 7fb394fa9640 1 --2- 172.21.15.7:0/124933700 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 0x7fb390136f80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00b4a0 tx=0x7fb38c00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.540+0000 7fb385ffb640 10 client.?.objecter ms_handle_connect 0x7fb3900b3e40 2024-01-21T03:20:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.540+0000 7fb385ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.547+0000 7fe2d525e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00a8b20 0x7fe2d0133e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.547+0000 7f16817fa640 1 -- 172.21.15.7:0/2317196430 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1674013470 con 0x7f168406f670 2024-01-21T03:20:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.547+0000 7f16817fa640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f167006c8b0 0x7f167006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7f1688d3a640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f167006c8b0 0x7f167006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7f1688d3a640 10 monclient: get_auth_request con 0x7f167006c8b0 auth_method 0 2024-01-21T03:20:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb385ffb640 1 -- 172.21.15.7:0/124933700 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c010610 con 0x7fb3900b3e40 2024-01-21T03:20:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb385ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb385ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb385ffb640 10 monclient: dump: 2024-01-21T03:20:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb385ffb640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb385ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb385ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb385ffb640 1 -- 172.21.15.7:0/124933700 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38c010770 con 0x7fb3900b3e40 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb385ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb385ffb640 1 -- 172.21.15.7:0/124933700 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c010b20 con 0x7fb3900b3e40 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb385ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb385ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb385ffb640 10 monclient: dump: 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.536 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb3877fe640 4 set_mon_vals no callback set 2024-01-21T03:20:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb397a0e640 5 monclient: authenticate success, global_id 4292 2024-01-21T03:20:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb397a0e640 10 monclient: _renew_subs 2024-01-21T03:20:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb397a0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb397a0e640 1 -- 172.21.15.7:0/124933700 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39013a530 con 0x7fb3900b3e40 2024-01-21T03:20:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb397a0e640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb397a0e640 10 monclient: _renew_subs 2024-01-21T03:20:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb397a0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb397a0e640 1 -- 172.21.15.7:0/124933700 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb390137ed0 con 0x7fb3900b3e40 2024-01-21T03:20:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fb397a0e640 1 librados: init done 2024-01-21T03:20:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fe2d525e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00a8b20 0x7fe2d0133e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45978/0 (socket says 172.21.15.7:45978) 2024-01-21T03:20:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fe2d525e640 1 -- 172.21.15.7:0/2944573048 learned_addr learned my addr 172.21.15.7:0/2944573048 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fe2d525e640 10 monclient(hunting): get_auth_request con 0x7fe2d00a8b20 auth_method 0 2024-01-21T03:20:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fe2d525e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fe2d525e640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.548+0000 7fe2d525e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.549+0000 7f1688d3a640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f167006c8b0 0x7f167006ed70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f167c02bf20 tx=0x7f167c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.549+0000 7f16817fa640 10 client.4299.objecter ms_handle_connect 0x7f167006c8b0 2024-01-21T03:20:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.549+0000 7fb397a0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.549+0000 7fb397a0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.549+0000 7fb397a0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.549+0000 7fb397a0e640 1 -- 172.21.15.7:0/124933700 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb390138150 con 0x7fb3900b3e40 2024-01-21T03:20:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.550+0000 7fe2d525e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.550+0000 7fe2d525e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.550+0000 7fe2d525e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.553+0000 7f16817fa640 1 -- 172.21.15.7:0/2317196430 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f167400ce30 con 0x7f168406f670 2024-01-21T03:20:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.553+0000 7f16817fa640 10 client.4299.objecter ms_dispatch 0x7f1684000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.553+0000 7f16817fa640 3 client.4299.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.553+0000 7f16817fa640 3 client.4299.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.553+0000 7f16817fa640 20 client.4299.objecter dump_active .. 0 homeless 2024-01-21T03:20:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.555+0000 7fe2d525e640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-01-21T03:20:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.555+0000 7fe2d525e640 1 -- 172.21.15.7:0/2944573048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2d00a9d10 msgr2=0x7fe2d0133460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.555+0000 7fe2d525e640 1 --2- 172.21.15.7:0/2944573048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2d00a9d10 0x7fe2d0133460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.555+0000 7fe2d525e640 1 -- 172.21.15.7:0/2944573048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2d00b2310 msgr2=0x7fe2d01d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.555+0000 7fe2d525e640 1 --2- 172.21.15.7:0/2944573048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2d00b2310 0x7fe2d01d7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.555+0000 7fe2d525e640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.555+0000 7fe2d525e640 1 monclient: found mon.noname-c 2024-01-21T03:20:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.555+0000 7fe2d525e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.555+0000 7fe2d525e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.555+0000 7fe2d525e640 1 -- 172.21.15.7:0/2944573048 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c8009b30 con 0x7fe2d00a8b20 2024-01-21T03:20:04.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.563+0000 7f00edffb640 1 -- 172.21.15.7:0/3891581075 <== osd.2 v2:172.21.15.7:6826/888884750 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f00dc075760 con 0x7f00dc072c90 2024-01-21T03:20:04.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.563+0000 7f00edffb640 10 client.4287.objecter ms_dispatch 0x7f00f8000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:20:04.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.563+0000 7f00edffb640 10 client.4287.objecter _finish_command 1 = system:0 2024-01-21T03:20:04.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.563+0000 7f00edffb640 20 client.4287.objecter put_session s=0x7f00dc0725c0 osd=2 4 2024-01-21T03:20:04.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.563+0000 7f00edffb640 15 client.4287.objecter _session_command_op_remove 2 1 2024-01-21T03:20:04.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.575+0000 7fb385ffb640 1 -- 172.21.15.7:0/124933700 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb38c010c80 con 0x7fb3900b3e40 2024-01-21T03:20:04.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.576+0000 7fb385ffb640 1 --2- 172.21.15.7:0/124933700 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb37406c8b0 0x7fb37406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.576+0000 7fb3957aa640 1 --2- 172.21.15.7:0/124933700 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb37406c8b0 0x7fb37406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.576+0000 7fb3957aa640 10 monclient: get_auth_request con 0x7fb37406c8b0 auth_method 0 2024-01-21T03:20:04.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.577+0000 7fb3957aa640 1 --2- 172.21.15.7:0/124933700 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb37406c8b0 0x7fb37406ed70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb388035000 tx=0x7fb38802e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.577+0000 7fb385ffb640 10 client.4292.objecter ms_handle_connect 0x7fb37406c8b0 2024-01-21T03:20:04.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.581+0000 7fe2d525e640 1 --2- 172.21.15.7:0/2944573048 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00a8b20 0x7fe2d0133e20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00c410 tx=0x7fe2cc00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.581+0000 7fe2c5ffb640 10 client.?.objecter ms_handle_connect 0x7fe2d00a8b20 2024-01-21T03:20:04.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.581+0000 7fe2c5ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.586+0000 7f09e42c9640 10 monclient: get_monmap_and_config 2024-01-21T03:20:04.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.586+0000 7f09e42c9640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.586+0000 7f09e42c9640 10 monclient: monmap: 2024-01-21T03:20:04.572 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.572 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.572 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.587527+0000 2024-01-21T03:20:04.572 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.587527+0000 2024-01-21T03:20:04.572 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.573 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.573 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:04.573 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:04.573 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:04.573 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:04.573 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:04.573 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.587+0000 7f09e42c9640 1 Processor -- start 2024-01-21T03:20:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.587+0000 7f09e42c9640 1 -- start start 2024-01-21T03:20:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.587+0000 7f09e42c9640 10 monclient: init 2024-01-21T03:20:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.588+0000 7f09e42c9640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.588+0000 7f09e42c9640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-01-21T03:20:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.588+0000 7f09e42c9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f09dc0a8b20 0x7f09dc145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.588+0000 7f09e42c9640 10 monclient(hunting): picked mon.noname-a con 0x7f09dc0a8b20 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.588+0000 7f09e42c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09dc0a9d10 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.588+0000 7f09e42c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09dc0b2310 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.588+0000 7f09e42c9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09dc1456f0 con 0x7f09dc0a9d10 2024-01-21T03:20:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.588+0000 7f09e42c9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09dc0a98c0 con 0x7f09dc0b2310 2024-01-21T03:20:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.588+0000 7f09e42c9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.588+0000 7f09e42c9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09dc1460c0 con 0x7f09dc0a8b20 2024-01-21T03:20:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.588+0000 7f09e42c9640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.588+0000 7f09e42c9640 10 monclient(hunting): authenticate will time out at 1350.904393s 2024-01-21T03:20:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.589+0000 7f00ff7e2640 10 client.4287.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T03:20:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.589+0000 7f00ff7e2640 15 client.4287.objecter _session_command_op_assign -1 2 2024-01-21T03:20:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.589+0000 7f00ff7e2640 20 client.4287.objecter _get_session s=0x7f00dc0725c0 osd=2 3 2024-01-21T03:20:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.589+0000 7f00ff7e2640 20 client.4287.objecter put_session s=0x7f00dc0725c0 osd=2 3 2024-01-21T03:20:04.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.589+0000 7f00ff7e2640 20 client.4287.objecter _get_session s=0x7f00dc0725c0 osd=2 3 2024-01-21T03:20:04.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.589+0000 7f00ff7e2640 15 client.4287.objecter _session_command_op_remove -1 2 2024-01-21T03:20:04.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.589+0000 7f00ff7e2640 20 client.4287.objecter get_session s=0x7f00dc0725c0 osd=2 3 2024-01-21T03:20:04.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.589+0000 7f00ff7e2640 15 client.4287.objecter _session_command_op_assign 2 2 2024-01-21T03:20:04.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.589+0000 7f00ff7e2640 20 client.4287.objecter put_session s=0x7f00dc0725c0 osd=2 4 2024-01-21T03:20:04.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.589+0000 7f00ff7e2640 10 client.4287.objecter _send_command 2 2024-01-21T03:20:04.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.589+0000 7f00ff7e2640 1 -- 172.21.15.7:0/3891581075 --> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f00f804e970 con 0x7f00dc072c90 2024-01-21T03:20:04.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.591+0000 7fb385ffb640 1 -- 172.21.15.7:0/124933700 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb38c058970 con 0x7fb3900b3e40 2024-01-21T03:20:04.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.591+0000 7fb385ffb640 10 client.4292.objecter ms_dispatch 0x7fb390000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:04.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.591+0000 7fb385ffb640 3 client.4292.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:04.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.591+0000 7fb385ffb640 3 client.4292.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:04.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.591+0000 7fb385ffb640 20 client.4292.objecter dump_active .. 0 homeless 2024-01-21T03:20:04.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.594+0000 7fe65e8c2640 10 monclient: get_monmap_and_config 2024-01-21T03:20:04.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.594+0000 7fe65e8c2640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.594+0000 7fe65e8c2640 10 monclient: monmap: 2024-01-21T03:20:04.580 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.580 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.581 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.595794+0000 2024-01-21T03:20:04.581 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.595794+0000 2024-01-21T03:20:04.581 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.581 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.581 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:04.581 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:04.581 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:04.581 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:04.581 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:04.581 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.595+0000 7fe65e8c2640 1 Processor -- start 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.595+0000 7fe65e8c2640 1 -- start start 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.595+0000 7fe65e8c2640 10 monclient: init 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.596+0000 7fe65e8c2640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.596+0000 7fe65e8c2640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.596+0000 7fe65e8c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6580a8b20 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.596+0000 7fe65e8c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6580a9d10 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.596+0000 7fe65e8c2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe6580b2310 0x7fe65813f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.596+0000 7fe65e8c2640 10 monclient(hunting): picked mon.noname-c con 0x7fe6580b2310 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.596+0000 7fe65e8c2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6581456f0 con 0x7fe6580a9d10 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.596+0000 7fe65e8c2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6580a98c0 con 0x7fe6580a8b20 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.596+0000 7fe65e8c2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.596+0000 7fe65e8c2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe658146290 con 0x7fe6580b2310 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.596+0000 7fe65e8c2640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.596+0000 7fe65e8c2640 10 monclient(hunting): authenticate will time out at 1350.912344s 2024-01-21T03:20:04.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.612+0000 7f00edffb640 1 -- 172.21.15.7:0/3891581075 <== osd.2 v2:172.21.15.7:6826/888884750 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f00f804e970 con 0x7f00dc072c90 2024-01-21T03:20:04.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.612+0000 7f00edffb640 10 client.4287.objecter ms_dispatch 0x7f00f8000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:20:04.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.612+0000 7f00edffb640 10 client.4287.objecter _finish_command 2 = system:0 2024-01-21T03:20:04.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.612+0000 7f00edffb640 20 client.4287.objecter put_session s=0x7f00dc0725c0 osd=2 4 2024-01-21T03:20:04.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.612+0000 7f00edffb640 15 client.4287.objecter _session_command_op_remove 2 2 2024-01-21T03:20:04.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 10 librados: watch_flush enter 2024-01-21T03:20:04.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 10 librados: watch_flush exit 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 10 client.4287.objecter close_session for osd.2 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 1 -- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] conn(0x7f00dc072c90 msgr2=0x7f00dc0750b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 1 --2- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] conn(0x7f00dc072c90 0x7f00dc0750b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 20 client.4287.objecter put_session s=0x7f00dc0725c0 osd=2 1 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 20 client.4287.objecter shutdown clearing up homeless session... 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 10 client.4287.objecter successfully canceled tick 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 1 -- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f00dc06c8b0 msgr2=0x7f00dc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 1 --2- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f00dc06c8b0 0x7f00dc06ed70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f00f4035000 tx=0x7f00f4030000 comp rx=0 tx=0).stop 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 10 monclient: shutdown 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 1 -- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00f8142b80 msgr2=0x7f00f81d73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 1 --2- 172.21.15.7:0/3891581075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00f8142b80 0x7f00f81d73c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f00f000b990 tx=0x7f00f000be50 comp rx=0 tx=0).stop 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00fdd7f640 1 -- 172.21.15.7:0/3891581075 reap_dead start 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 1 -- 172.21.15.7:0/3891581075 shutdown_connections 2024-01-21T03:20:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 1 -- 172.21.15.7:0/3891581075 >> 172.21.15.7:0/3891581075 conn(0x7f00f80b3a80 msgr2=0x7f00f813bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 1 -- 172.21.15.7:0/3891581075 shutdown_connections 2024-01-21T03:20:04.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 1 -- 172.21.15.7:0/3891581075 wait complete. 2024-01-21T03:20:04.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.613+0000 7f00ff7e2640 1 librados: shutdown 2024-01-21T03:20:04.610 INFO:teuthology.orchestra.run.smithi007.stdout:90194313224 2024-01-21T03:20:04.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-21T03:20:04.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.706+0000 7ff23759e640 1 --2- 172.21.15.7:0/397666239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2200052a0 0x7ff220014190 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009760 tx=0x7ff2200057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=982475a222be9014 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.723+0000 7fabe3fff640 1 -- 172.21.15.7:0/1199614872 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fabe400c670 con 0x7fabf0035240 2024-01-21T03:20:04.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.724+0000 7fabe3fff640 1 --2- 172.21.15.7:0/1199614872 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fabdc06c8e0 0x7fabdc06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.724+0000 7fabf77fe640 1 --2- 172.21.15.7:0/1199614872 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fabdc06c8e0 0x7fabdc06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.724+0000 7fabf77fe640 10 monclient: get_auth_request con 0x7fabdc06c8e0 auth_method 0 2024-01-21T03:20:04.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.724+0000 7fabf77fe640 1 --2- 172.21.15.7:0/1199614872 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fabdc06c8e0 0x7fabdc06eda0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fabec035000 tx=0x7fabec030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.724+0000 7fabe3fff640 10 client.4330.objecter ms_handle_connect 0x7fabdc06c8e0 2024-01-21T03:20:04.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.726+0000 7fabe3fff640 1 -- 172.21.15.7:0/1199614872 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fabe4058de0 con 0x7fabf0035240 2024-01-21T03:20:04.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.726+0000 7fabe3fff640 10 client.4330.objecter ms_dispatch 0x7fabf8000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:04.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.726+0000 7fabe3fff640 3 client.4330.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:04.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.726+0000 7fabe3fff640 3 client.4330.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:04.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.726+0000 7fabe3fff640 20 client.4330.objecter dump_active .. 0 homeless 2024-01-21T03:20:04.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.736+0000 7fabe3fff640 1 -- 172.21.15.7:0/1199614872 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fabe4054a20 con 0x7fabf0035240 2024-01-21T03:20:04.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.736+0000 7fabe3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.736+0000 7fabe3fff640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:04.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.743+0000 7fe65ce5f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe6580b2310 0x7fe65813f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.743+0000 7fe65ce5f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe6580b2310 0x7fe65813f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36730/0 (socket says 172.21.15.7:36730) 2024-01-21T03:20:04.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.743+0000 7fe65ce5f640 1 -- 172.21.15.7:0/303673666 learned_addr learned my addr 172.21.15.7:0/303673666 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.743+0000 7fe65ce5f640 10 monclient(hunting): get_auth_request con 0x7fe6580b2310 auth_method 0 2024-01-21T03:20:04.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.743+0000 7fe65ce5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.743+0000 7fe65ce5f640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.743+0000 7fe65ce5f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.744+0000 7fe65ce5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.744+0000 7fe65ce5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.744+0000 7fe65ce5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.747+0000 7fe65ce5f640 10 monclient(hunting): handle_auth_done global_id 4333 payload 274 2024-01-21T03:20:04.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.747+0000 7fe65ce5f640 1 -- 172.21.15.7:0/303673666 >> v1:172.21.15.177:6789/0 conn(0x7fe6580a8b20 legacy=0x7fe658145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.747+0000 7fe65ce5f640 1 -- 172.21.15.7:0/303673666 >> v1:172.21.15.7:6789/0 conn(0x7fe6580a9d10 legacy=0x7fe65813f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.747+0000 7fe65ce5f640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.747+0000 7fe65ce5f640 1 monclient: found mon.noname-c 2024-01-21T03:20:04.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.747+0000 7fe65ce5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.747+0000 7fe65ce5f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.747+0000 7fe65ce5f640 1 -- 172.21.15.7:0/303673666 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6581463d0 con 0x7fe6580b2310 2024-01-21T03:20:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c032600 con 0x7ff2200052a0 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff235d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff235d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff235d9b640 10 monclient: dump: 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22c02e050 con 0x7ff2200052a0 2024-01-21T03:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff235d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff23d061640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff235d9b640 1 -- 172.21.15.7:0/397666239 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0329a0 con 0x7ff2200052a0 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff235d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff23d061640 4 set_mon_vals no callback set 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff235d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff235d9b640 10 monclient: dump: 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.737 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff23d061640 10 monclient: shutdown 2024-01-21T03:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff23d061640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff23d061640 1 -- 172.21.15.7:0/397666239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2200052a0 msgr2=0x7ff220014190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff23d061640 1 --2- 172.21.15.7:0/397666239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2200052a0 0x7ff220014190 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009760 tx=0x7ff2200057f0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff23759e640 1 -- 172.21.15.7:0/397666239 reap_dead start 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff23d061640 1 -- 172.21.15.7:0/397666239 shutdown_connections 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff23d061640 1 -- 172.21.15.7:0/397666239 >> 172.21.15.7:0/397666239 conn(0x7ff230006610 msgr2=0x7ff230006a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff23d061640 1 -- 172.21.15.7:0/397666239 shutdown_connections 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.749+0000 7ff23d061640 1 -- 172.21.15.7:0/397666239 wait complete. 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.751+0000 7ff23d061640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.751+0000 7ff23d061640 10 monclient: monmap: 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.751+0000 7ff23d061640 1 librados: starting msgr at 2024-01-21T03:20:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.751+0000 7ff23d061640 1 librados: starting objecter 2024-01-21T03:20:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.751+0000 7ff23d061640 1 Processor -- start 2024-01-21T03:20:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.751+0000 7ff23d061640 1 -- start start 2024-01-21T03:20:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.751+0000 7ff23d061640 1 librados: setting wanted keys 2024-01-21T03:20:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.751+0000 7ff23d061640 1 librados: calling monclient init 2024-01-21T03:20:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.751+0000 7ff23d061640 10 monclient: init 2024-01-21T03:20:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7f09e2866640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f09dc0b2310 0x7f09dc14c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:44416/0 (socket says 172.21.15.7:44416) 2024-01-21T03:20:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7f09e2866640 1 -- 172.21.15.7:0/719424927 learned_addr learned my addr 172.21.15.7:0/719424927 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2200052a0 0x7ff2300c0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 10 monclient(hunting): picked mon.noname-a con 0x7ff2200052a0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2300c0840 0x7ff2300be640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 10 monclient(hunting): picked mon.noname-b con 0x7ff2300c0840 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2300c20b0 0x7ff2300beb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 10 monclient(hunting): picked mon.noname-c con 0x7ff2300c20b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2300bf0c0 con 0x7ff2200052a0 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2300c1a90 con 0x7ff2300c0840 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23002ddb0 con 0x7ff2300c20b0 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.752+0000 7ff23d061640 10 monclient(hunting): authenticate will time out at 1351.068719s 2024-01-21T03:20:04.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.753+0000 7fe65ce5f640 1 --2- 172.21.15.7:0/303673666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6580b2310 0x7fe65813f790 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe650009e50 tx=0x7fe65002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c39b9a28aae0a257 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe656ffd640 1 -- 172.21.15.7:0/303673666 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65002ca80 con 0x7fe6580b2310 2024-01-21T03:20:04.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe656ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe656ffd640 10 monclient: dump: 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe656ffd640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe656ffd640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe656ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe656ffd640 1 -- 172.21.15.7:0/303673666 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65002cbe0 con 0x7fe6580b2310 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe65e8c2640 5 monclient: authenticate success, global_id 4333 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe65e8c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe656ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe65e8c2640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe656ffd640 1 -- 172.21.15.7:0/303673666 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6500343f0 con 0x7fe6580b2310 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe65e8c2640 4 set_mon_vals no callback set 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe656ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.760+0000 7fe656ffd640 10 monclient: dump: 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775930502 0 0) 0x7f09dc0a98c0 con 0x7f09dc0b2310 2024-01-21T03:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7f09e1063640 10 monclient(hunting): _negotiate 2024-01-21T03:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7f09e1063640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7f09e1063640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7f09e1063640 10 monclient(hunting): my global_id is 4336 2024-01-21T03:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c8003520 con 0x7f09dc0b2310 2024-01-21T03:20:04.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7fe65e8c2640 10 monclient: shutdown 2024-01-21T03:20:04.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7fe65e8c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7fe65e8c2640 1 -- 172.21.15.7:0/303673666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6580b2310 msgr2=0x7fe65813f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7fe65e8c2640 1 --2- 172.21.15.7:0/303673666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6580b2310 0x7fe65813f790 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe650009e50 tx=0x7fe65002bee0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7fe65e8c2640 1 -- 172.21.15.7:0/303673666 shutdown_connections 2024-01-21T03:20:04.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7fe65e8c2640 1 --2- 172.21.15.7:0/303673666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6580b2310 0x7fe65813f790 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7fe65e8c2640 1 -- 172.21.15.7:0/303673666 >> 172.21.15.7:0/303673666 conn(0x7fe658066fb0 msgr2=0x7fe6580b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7ff23659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2300c0840 0x7ff2300be640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7fe65e8c2640 1 -- 172.21.15.7:0/303673666 shutdown_connections 2024-01-21T03:20:04.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.761+0000 7fe65e8c2640 1 -- 172.21.15.7:0/303673666 wait complete. 2024-01-21T03:20:04.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.762+0000 7ff23659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2300c0840 0x7ff2300be640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36740/0 (socket says 172.21.15.7:36740) 2024-01-21T03:20:04.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.762+0000 7ff23659c640 1 -- 172.21.15.7:0/813608811 learned_addr learned my addr 172.21.15.7:0/813608811 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.762+0000 7ff23659c640 10 monclient(hunting): get_auth_request con 0x7ff2300c0840 auth_method 0 2024-01-21T03:20:04.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.762+0000 7ff23659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.762+0000 7ff23659c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.762+0000 7ff23659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.762+0000 7fe65e8c2640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.762+0000 7fe65e8c2640 10 monclient: monmap: 2024-01-21T03:20:04.753 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.753 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.753 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.753 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.753 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.753 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.753 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.753 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.753 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.753 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.763+0000 7ff23659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.763+0000 7ff23659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.763+0000 7ff23659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.763+0000 7fe65e8c2640 1 librados: starting msgr at 2024-01-21T03:20:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.763+0000 7fe65e8c2640 1 librados: starting objecter 2024-01-21T03:20:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.763+0000 7fe65e8c2640 1 Processor -- start 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.763+0000 7fe65e8c2640 1 -- start start 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.763+0000 7fe65e8c2640 1 librados: setting wanted keys 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.763+0000 7fe65e8c2640 1 librados: calling monclient init 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.763+0000 7fe65e8c2640 10 monclient: init 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6580a8b20 0x7fe65813ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 10 monclient(hunting): picked mon.noname-a con 0x7fe6580a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6580a9d10 0x7fe65813bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 10 monclient(hunting): picked mon.noname-b con 0x7fe6580a9d10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6580b2310 0x7fe658141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 10 monclient(hunting): picked mon.noname-c con 0x7fe6580b2310 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6581415d0 con 0x7fe6580a8b20 2024-01-21T03:20:04.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe658141840 con 0x7fe6580a9d10 2024-01-21T03:20:04.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe658141b20 con 0x7fe6580b2310 2024-01-21T03:20:04.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.764+0000 7fe65e8c2640 10 monclient(hunting): authenticate will time out at 1351.080058s 2024-01-21T03:20:04.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.766+0000 7ff23659c640 10 monclient(hunting): handle_auth_done global_id 4339 payload 824 2024-01-21T03:20:04.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.766+0000 7ff23659c640 1 -- 172.21.15.7:0/813608811 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2300c20b0 msgr2=0x7ff2300beb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.766+0000 7ff23659c640 1 --2- 172.21.15.7:0/813608811 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2300c20b0 0x7ff2300beb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.766+0000 7ff23659c640 1 -- 172.21.15.7:0/813608811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2200052a0 msgr2=0x7ff2300c0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.766+0000 7ff23659c640 1 --2- 172.21.15.7:0/813608811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2200052a0 0x7ff2300c0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.766+0000 7ff23659c640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.766+0000 7ff23659c640 1 monclient: found mon.noname-b 2024-01-21T03:20:04.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.766+0000 7ff23659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.766+0000 7ff23659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.766+0000 7ff23659c640 1 -- 172.21.15.7:0/813608811 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2300bfb40 con 0x7ff2300c0840 2024-01-21T03:20:04.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.769+0000 7ff23659c640 1 --2- 172.21.15.7:0/813608811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2300c0840 0x7ff2300be640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff238050150 tx=0x7ff238067a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.769+0000 7ff21f7fe640 10 client.?.objecter ms_handle_connect 0x7ff2300c0840 2024-01-21T03:20:04.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.769+0000 7ff21f7fe640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.769+0000 7f29d959e640 10 monclient: get_monmap_and_config 2024-01-21T03:20:04.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.769+0000 7f29d959e640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.769+0000 7f29d959e640 10 monclient: monmap: 2024-01-21T03:20:04.760 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.760 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.760 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:04.770569+0000 2024-01-21T03:20:04.760 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:04.770569+0000 2024-01-21T03:20:04.760 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 1 Processor -- start 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 1 -- start start 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 10 monclient: init 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f29d40a8b20 0x7f29d4145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 10 monclient(hunting): picked mon.noname-b con 0x7f29d40a8b20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f29d40a9d10 0x7f29d413f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 10 monclient(hunting): picked mon.noname-c con 0x7f29d40a9d10 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29d40b2310 0x7f29d413f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 10 monclient(hunting): picked mon.noname-a con 0x7f29d40b2310 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d41456f0 con 0x7f29d40b2310 2024-01-21T03:20:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d40a98c0 con 0x7f29d40a9d10 2024-01-21T03:20:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.770+0000 7f29d959e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d413fb00 con 0x7f29d40a8b20 2024-01-21T03:20:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.771+0000 7f29d959e640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.771+0000 7f29d959e640 10 monclient(hunting): authenticate will time out at 1351.086764s 2024-01-21T03:20:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.774+0000 7fe6577fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6580a9d10 0x7fe65813bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7fe6577fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6580a9d10 0x7fe65813bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36752/0 (socket says 172.21.15.7:36752) 2024-01-21T03:20:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7fe6577fe640 1 -- 172.21.15.7:0/1517079039 learned_addr learned my addr 172.21.15.7:0/1517079039 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7fe6577fe640 10 monclient(hunting): get_auth_request con 0x7fe6580a9d10 auth_method 0 2024-01-21T03:20:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7fe6577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7fe6577fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7fe6577fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 672629159 0 0) 0x7f09c8003520 con 0x7f09dc0b2310 2024-01-21T03:20:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 >> v2:172.21.15.7:3300/0 conn(0x7f09dc0a8b20 msgr2=0x7f09dc145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 --2- 172.21.15.7:0/719424927 >> v2:172.21.15.7:3300/0 conn(0x7f09dc0a8b20 0x7f09dc145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 >> v1:172.21.15.7:6789/0 conn(0x7f09dc0a9d10 legacy=0x7f09dc13f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f29cbfff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f29d40a9d10 0x7f29d413f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 monclient: found mon.noname-c-legacy 2024-01-21T03:20:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.177:6789/0 2024-01-21T03:20:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc146200 con 0x7f09dc0b2310 2024-01-21T03:20:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 749591385 0 0) 0x7f09d40019f0 con 0x7f09dc0b2310 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient: dump: 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] but i'm connected to v1:172.21.15.177:6789/0, reconnecting 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 >> v1:172.21.15.177:6789/0 conn(0x7f09dc0b2310 legacy=0x7f09dc14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 --2- 172.21.15.7:0/719424927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c80039c0 0x7f09c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient(hunting): picked mon.a con 0x7f09c80039c0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 --2- 172.21.15.7:0/719424927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09c80047c0 0x7f09c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient(hunting): picked mon.b con 0x7f09c80047c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 --2- 172.21.15.7:0/719424927 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09c8005330 0x7f09c800b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient(hunting): picked mon.c con 0x7f09c8005330 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc1460c0 con 0x7f09c80039c0 2024-01-21T03:20:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c800c280 con 0x7f09c80047c0 2024-01-21T03:20:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c800c440 con 0x7f09c8005330 2024-01-21T03:20:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.775+0000 7f09e1063640 10 monclient(hunting): _finish_auth 0 2024-01-21T03:20:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7fe6577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7fe6577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7fe6577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff21f7fe640 1 -- 172.21.15.7:0/813608811 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff238093440 con 0x7ff2300c0840 2024-01-21T03:20:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff21f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff21f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff21f7fe640 10 monclient: dump: 2024-01-21T03:20:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff21f7fe640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff21f7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff21f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff21f7fe640 1 -- 172.21.15.7:0/813608811 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff238072050 con 0x7ff2300c0840 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff21f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff21f7fe640 1 -- 172.21.15.7:0/813608811 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2380937e0 con 0x7ff2300c0840 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff21f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff21f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff21f7fe640 10 monclient: dump: 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff23d061640 5 monclient: authenticate success, global_id 4339 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff23d061640 10 monclient: _renew_subs 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff23d061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff23d061640 1 -- 172.21.15.7:0/813608811 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff230158ce0 con 0x7ff2300c0840 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff23559a640 4 set_mon_vals no callback set 2024-01-21T03:20:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff23d061640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.777+0000 7f29cbfff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f29d40a9d10 0x7f29d413f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36754/0 (socket says 172.21.15.7:36754) 2024-01-21T03:20:04.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.777+0000 7f29cbfff640 1 -- 172.21.15.7:0/3109762236 learned_addr learned my addr 172.21.15.7:0/3109762236 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.777+0000 7f29cbfff640 10 monclient(hunting): get_auth_request con 0x7f29d40a9d10 auth_method 0 2024-01-21T03:20:04.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.777+0000 7f29cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.777+0000 7f29cbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.777+0000 7f29cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.777+0000 7f29cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.777+0000 7f29cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.777+0000 7f29cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff23d061640 10 monclient: _renew_subs 2024-01-21T03:20:04.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff23d061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff23d061640 1 -- 172.21.15.7:0/813608811 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff230159170 con 0x7ff2300c0840 2024-01-21T03:20:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.776+0000 7ff23d061640 1 librados: init done 2024-01-21T03:20:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.777+0000 7ff23d061640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.777+0000 7ff23d061640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.777+0000 7ff23d061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.777+0000 7ff23d061640 1 -- 172.21.15.7:0/813608811 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff230008630 con 0x7ff2300c0840 2024-01-21T03:20:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.780+0000 7fe6577fe640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2024-01-21T03:20:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.781+0000 7fe6577fe640 1 -- 172.21.15.7:0/1517079039 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6580b2310 msgr2=0x7fe658141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.781+0000 7fe6577fe640 1 --2- 172.21.15.7:0/1517079039 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6580b2310 0x7fe658141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.781+0000 7fe6577fe640 1 -- 172.21.15.7:0/1517079039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6580a8b20 msgr2=0x7fe65813ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.781+0000 7fe6577fe640 1 --2- 172.21.15.7:0/1517079039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6580a8b20 0x7fe65813ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.781+0000 7fe6577fe640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.781+0000 7fe6577fe640 1 monclient: found mon.noname-b 2024-01-21T03:20:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.781+0000 7fe6577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.781+0000 7fe6577fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.781+0000 7fe6577fe640 1 -- 172.21.15.7:0/1517079039 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe650009b30 con 0x7fe6580a9d10 2024-01-21T03:20:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.783+0000 7f29cbfff640 10 monclient(hunting): handle_auth_done global_id 4345 payload 274 2024-01-21T03:20:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.783+0000 7f29cbfff640 1 -- 172.21.15.7:0/3109762236 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f29d40a8b20 msgr2=0x7f29d4145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.783+0000 7f29cbfff640 1 --2- 172.21.15.7:0/3109762236 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f29d40a8b20 0x7f29d4145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.783+0000 7f29cbfff640 1 -- 172.21.15.7:0/3109762236 >> v2:172.21.15.7:3300/0 conn(0x7f29d40b2310 msgr2=0x7f29d413f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.783+0000 7f29cbfff640 1 --2- 172.21.15.7:0/3109762236 >> v2:172.21.15.7:3300/0 conn(0x7f29d40b2310 0x7f29d413f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.783+0000 7f29cbfff640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.783+0000 7f29cbfff640 1 monclient: found mon.noname-c 2024-01-21T03:20:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.783+0000 7f29cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.783+0000 7f29cbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:20:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.783+0000 7f29cbfff640 1 -- 172.21.15.7:0/3109762236 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d41460c0 con 0x7f29d40a9d10 2024-01-21T03:20:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.792+0000 7f09e1864640 1 --2- 172.21.15.7:0/719424927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09c80047c0 0x7f09c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.792+0000 7f09e1864640 10 monclient(hunting): get_auth_request con 0x7f09c80047c0 auth_method 0 2024-01-21T03:20:04.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.792+0000 7f09e1864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.792+0000 7f09e1864640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.792+0000 7f09e1864640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:20:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.847+0000 7fb385ffb640 1 -- 172.21.15.7:0/124933700 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb38c0545b0 con 0x7fb3900b3e40 2024-01-21T03:20:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.847+0000 7fb385ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.847+0000 7fb385ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c5ffb640 1 -- 172.21.15.7:0/2944573048 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc0035f0 con 0x7fe2d00a8b20 2024-01-21T03:20:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c5ffb640 10 monclient: dump: 2024-01-21T03:20:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c5ffb640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c5ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c5ffb640 1 -- 172.21.15.7:0/2944573048 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2cc003750 con 0x7fe2d00a8b20 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c5ffb640 1 -- 172.21.15.7:0/2944573048 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc003b40 con 0x7fe2d00a8b20 2024-01-21T03:20:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2d74c2640 5 monclient: authenticate success, global_id 4295 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2d74c2640 10 monclient: _renew_subs 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2d74c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2d74c2640 1 -- 172.21.15.7:0/2944573048 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d01339a0 con 0x7fe2d00a8b20 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c77fe640 4 set_mon_vals no callback set 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2c5ffb640 10 monclient: dump: 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.839 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2d74c2640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2d74c2640 10 monclient: _renew_subs 2024-01-21T03:20:04.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2d74c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2d74c2640 1 -- 172.21.15.7:0/2944573048 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d01d7ea0 con 0x7fe2d00a8b20 2024-01-21T03:20:04.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.850+0000 7fe2d74c2640 1 librados: init done 2024-01-21T03:20:04.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.851+0000 7fe2d74c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.851+0000 7fe2d74c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.851+0000 7fe2d74c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:04.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.851+0000 7fe2d74c2640 1 -- 172.21.15.7:0/2944573048 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d01408e0 con 0x7fe2d00a8b20 2024-01-21T03:20:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.860+0000 7f09e2866640 1 --2- 172.21.15.7:0/719424927 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09c8005330 0x7f09c800b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.863+0000 7f16817fa640 1 -- 172.21.15.7:0/2317196430 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1674054ef0 con 0x7f168406f670 2024-01-21T03:20:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.863+0000 7f16817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.863+0000 7f16817fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:04.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.866+0000 7f09e2065640 1 --2- 172.21.15.7:0/719424927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c80039c0 0x7f09c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.869+0000 7f09e2065640 10 monclient(hunting): get_auth_request con 0x7f09c80039c0 auth_method 0 2024-01-21T03:20:04.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.869+0000 7f09e2065640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.869+0000 7f09e2065640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.869+0000 7f09e2065640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:20:04.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.875+0000 7f09e2065640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.875+0000 7f09e2065640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.875+0000 7f09e2065640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T03:20:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2065640 10 monclient(hunting): handle_auth_done global_id 4336 payload 293 2024-01-21T03:20:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2065640 1 -- 172.21.15.7:0/719424927 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09c8005330 msgr2=0x7f09c800b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2065640 1 --2- 172.21.15.7:0/719424927 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09c8005330 0x7f09c800b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2065640 1 -- 172.21.15.7:0/719424927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09c80047c0 msgr2=0x7f09c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2065640 1 --2- 172.21.15.7:0/719424927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09c80047c0 0x7f09c8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2065640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2065640 1 monclient: found mon.a 2024-01-21T03:20:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2065640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2866640 1 -- 172.21.15.7:0/719424927 reap_dead start 2024-01-21T03:20:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2065640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2065640 1 -- 172.21.15.7:0/719424927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c8003520 con 0x7f09c80039c0 2024-01-21T03:20:04.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2065640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2065640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e2065640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e42c9640 5 monclient: authenticate success, global_id 4336 2024-01-21T03:20:04.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.878+0000 7f09e42c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:04.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.881+0000 7f09e2065640 1 --2- 172.21.15.7:0/719424927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c80039c0 0x7f09c8003dc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f09d8009860 tx=0x7f09c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd04a0f541f1d0e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.887+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8032600 con 0x7f09c80039c0 2024-01-21T03:20:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.887+0000 7f09e1063640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.887+0000 7f09e1063640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.887+0000 7f09e1063640 10 monclient: dump: 2024-01-21T03:20:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.874 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.887+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d802e050 con 0x7f09c80039c0 2024-01-21T03:20:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.887+0000 7f09e1063640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.887+0000 7f09e1063640 1 -- 172.21.15.7:0/719424927 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d80329c0 con 0x7f09c80039c0 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.887+0000 7f09e1063640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.887+0000 7f09e1063640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.887+0000 7f09e1063640 10 monclient: dump: 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.887+0000 7f09e42c9640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:04.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.887+0000 7f09e42c9640 4 set_mon_vals no callback set 2024-01-21T03:20:04.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.888+0000 7f09e42c9640 10 monclient: shutdown 2024-01-21T03:20:04.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.888+0000 7f09e42c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:04.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.888+0000 7f09e42c9640 1 -- 172.21.15.7:0/719424927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c80039c0 msgr2=0x7f09c8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.888+0000 7f09e42c9640 1 --2- 172.21.15.7:0/719424927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c80039c0 0x7f09c8003dc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f09d8009860 tx=0x7f09c80057f0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.888+0000 7f09e42c9640 1 -- 172.21.15.7:0/719424927 shutdown_connections 2024-01-21T03:20:04.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.888+0000 7f09e42c9640 1 --2- 172.21.15.7:0/719424927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c80039c0 0x7f09c8003dc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.888+0000 7f09e42c9640 1 -- 172.21.15.7:0/719424927 >> 172.21.15.7:0/719424927 conn(0x7f09dc066fb0 msgr2=0x7f09dc0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:04.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.889+0000 7f09e42c9640 1 -- 172.21.15.7:0/719424927 shutdown_connections 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.889+0000 7f09e42c9640 1 -- 172.21.15.7:0/719424927 wait complete. 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.890+0000 7f09e42c9640 10 monclient: build_initial_monmap 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.890+0000 7f09e42c9640 10 monclient: monmap: 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.890+0000 7f09e42c9640 1 librados: starting msgr at 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.890+0000 7f09e42c9640 1 librados: starting objecter 2024-01-21T03:20:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.890+0000 7f09e42c9640 1 Processor -- start 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.890+0000 7f09e42c9640 1 -- start start 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.890+0000 7f09e42c9640 1 librados: setting wanted keys 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.890+0000 7f09e42c9640 1 librados: calling monclient init 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.890+0000 7f09e42c9640 10 monclient: init 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.891+0000 7f09e42c9640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.891+0000 7f09e42c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.891+0000 7f09e42c9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09c80039c0 0x7f09dc1dac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.891+0000 7f09e42c9640 10 monclient(hunting): picked mon.noname-c con 0x7f09c80039c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.892+0000 7f09e42c9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09dc1db150 0x7f09dc0b0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.892+0000 7f09e42c9640 10 monclient(hunting): picked mon.noname-b con 0x7f09dc1db150 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.892+0000 7f09e42c9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc13a8c0 0x7f09dc0b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.892+0000 7f09e42c9640 10 monclient(hunting): picked mon.noname-a con 0x7f09dc13a8c0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.892+0000 7f09e42c9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.892+0000 7f09e42c9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc1db6e0 con 0x7f09dc13a8c0 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.892+0000 7f09e42c9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.892+0000 7f09e42c9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc0abda0 con 0x7f09dc1db150 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.892+0000 7f09e42c9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.892+0000 7f09e42c9640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc0b1290 con 0x7f09c80039c0 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.892+0000 7f09e42c9640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.892+0000 7f09e2866640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc13a8c0 0x7f09dc0b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.892+0000 7f09e42c9640 10 monclient(hunting): authenticate will time out at 1351.208734s 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.893+0000 7f09e2866640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc13a8c0 0x7f09dc0b0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44358/0 (socket says 172.21.15.7:44358) 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.893+0000 7f09e2866640 1 -- 172.21.15.7:0/2635695433 learned_addr learned my addr 172.21.15.7:0/2635695433 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.893+0000 7f09e2866640 10 monclient(hunting): get_auth_request con 0x7f09dc13a8c0 auth_method 0 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.893+0000 7f09e2866640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.893+0000 7f09e2866640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.893+0000 7f09e2866640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.894+0000 7f09e2866640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.894+0000 7f09e2866640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.894+0000 7f09e2866640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:04.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.898+0000 7f09e2866640 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2024-01-21T03:20:04.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.898+0000 7f09e2866640 1 -- 172.21.15.7:0/2635695433 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09c80039c0 msgr2=0x7f09dc1dac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.898+0000 7f09e2866640 1 --2- 172.21.15.7:0/2635695433 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09c80039c0 0x7f09dc1dac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.898+0000 7f09e2866640 1 -- 172.21.15.7:0/2635695433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09dc1db150 msgr2=0x7f09dc0b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:04.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.898+0000 7f09e2866640 1 --2- 172.21.15.7:0/2635695433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09dc1db150 0x7f09dc0b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:04.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.898+0000 7f09e2866640 10 monclient: _finish_hunting 0 2024-01-21T03:20:04.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.898+0000 7f09e2866640 1 monclient: found mon.noname-a 2024-01-21T03:20:04.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.898+0000 7f09e2866640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:04.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.898+0000 7f09e2866640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.898+0000 7f09e2866640 1 -- 172.21.15.7:0/2635695433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc142590 con 0x7f09dc13a8c0 2024-01-21T03:20:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.902+0000 7f09e2866640 1 --2- 172.21.15.7:0/2635695433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc13a8c0 0x7f09dc0b0d50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0b41b0 tx=0x7f09d400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.902+0000 7f09d2ffd640 10 client.?.objecter ms_handle_connect 0x7f09dc13a8c0 2024-01-21T03:20:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.902+0000 7f09d2ffd640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09d2ffd640 1 -- 172.21.15.7:0/2635695433 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d4004d60 con 0x7f09dc13a8c0 2024-01-21T03:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09d2ffd640 10 monclient: dump: 2024-01-21T03:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09d2ffd640 10 monclient: _finish_auth 0 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09d2ffd640 10 monclient: _check_auth_tickets 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09e42c9640 5 monclient: authenticate success, global_id 4302 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09e42c9640 10 monclient: _renew_subs 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09e42c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09e42c9640 1 -- 172.21.15.7:0/2635695433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09dc142780 con 0x7f09dc13a8c0 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09e42c9640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09e42c9640 10 monclient: _renew_subs 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09e42c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09e42c9640 1 -- 172.21.15.7:0/2635695433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09dc142c40 con 0x7f09dc13a8c0 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.906+0000 7f09e42c9640 1 librados: init done 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.907+0000 7f09e42c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.907+0000 7f09e42c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.907+0000 7f09e42c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.907+0000 7f09e42c9640 1 -- 172.21.15.7:0/2635695433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09dc04e970 con 0x7f09dc13a8c0 2024-01-21T03:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.907+0000 7f09d2ffd640 1 -- 172.21.15.7:0/2635695433 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d400bb80 con 0x7f09dc13a8c0 2024-01-21T03:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.907+0000 7f09d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.907+0000 7f09e0862640 4 set_mon_vals no callback set 2024-01-21T03:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.908+0000 7f09d2ffd640 1 -- 172.21.15.7:0/2635695433 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d4007070 con 0x7f09dc13a8c0 2024-01-21T03:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.908+0000 7f09d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.908+0000 7f09d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:04.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:04.908+0000 7f09d2ffd640 10 monclient: dump: 2024-01-21T03:20:04.897 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:04.897 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:04.897 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.897 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:04.897 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:04.897 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:04.897 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:04.897 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:04.897 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:04.897 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:05.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.020+0000 7f29cbfff640 1 --2- 172.21.15.7:0/3109762236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d40a9d10 0x7f29d413f080 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f29d40b54d0 tx=0x7f29bc02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e11eed93e6865cf server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:05.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.033+0000 7ff21f7fe640 1 -- 172.21.15.7:0/813608811 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ff2380923e0 con 0x7ff2300c0840 2024-01-21T03:20:05.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.034+0000 7fe6577fe640 1 --2- 172.21.15.7:0/1517079039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6580a9d10 0x7fe65813bb70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00c400 tx=0x7fe64c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:05.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.034+0000 7fe654ff9640 10 client.?.objecter ms_handle_connect 0x7fe6580a9d10 2024-01-21T03:20:05.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.034+0000 7fe654ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:05.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.034+0000 7ff21f7fe640 1 --2- 172.21.15.7:0/813608811 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ff22406c890 0x7ff22406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:05.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.034+0000 7ff236d9d640 1 --2- 172.21.15.7:0/813608811 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ff22406c890 0x7ff22406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:05.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.034+0000 7ff236d9d640 10 monclient: get_auth_request con 0x7ff22406c890 auth_method 0 2024-01-21T03:20:05.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.035+0000 7ff236d9d640 1 --2- 172.21.15.7:0/813608811 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ff22406c890 0x7ff22406ed50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff228005ae0 tx=0x7ff228001d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:05.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.035+0000 7ff21f7fe640 10 client.4339.objecter ms_handle_connect 0x7ff22406c890 2024-01-21T03:20:05.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.038+0000 7fabe1ffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T03:20:05.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.038+0000 7fabe1ffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T03:20:05.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.038+0000 7fabe1ffb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:05.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.038+0000 7fabe1ffb640 1 -- 172.21.15.7:0/1199614872 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fabd8005600 con 0x7fabf0035240 2024-01-21T03:20:05.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.041+0000 7ff21f7fe640 1 -- 172.21.15.7:0/813608811 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff2380d2ce0 con 0x7ff2300c0840 2024-01-21T03:20:05.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.041+0000 7ff21f7fe640 10 client.4339.objecter ms_dispatch 0x7ff238000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:05.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.041+0000 7ff21f7fe640 3 client.4339.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:05.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.041+0000 7ff21f7fe640 3 client.4339.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:05.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.041+0000 7ff21f7fe640 20 client.4339.objecter dump_active .. 0 homeless 2024-01-21T03:20:05.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.050+0000 7ff21f7fe640 1 -- 172.21.15.7:0/813608811 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ff2380ce920 con 0x7ff2300c0840 2024-01-21T03:20:05.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.050+0000 7ff21f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:05.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.050+0000 7ff21f7fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:05.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d27fc640 1 -- 172.21.15.7:0/3109762236 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc031870 con 0x7f29d40a9d10 2024-01-21T03:20:05.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:05.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:05.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d27fc640 10 monclient: dump: 2024-01-21T03:20:05.041 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:05.041 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:05.041 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:05.041 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:05.041 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:05.041 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d27fc640 10 monclient: _finish_auth 0 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d27fc640 10 monclient: _check_auth_tickets 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d27fc640 1 -- 172.21.15.7:0/3109762236 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29bc0319d0 con 0x7f29d40a9d10 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d959e640 5 monclient: authenticate success, global_id 4345 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d959e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d27fc640 1 -- 172.21.15.7:0/3109762236 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc031d60 con 0x7f29d40a9d10 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d959e640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.053+0000 7f29d959e640 4 set_mon_vals no callback set 2024-01-21T03:20:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.054+0000 7f29d959e640 10 monclient: shutdown 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.054+0000 7f29d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.054+0000 7f29d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.054+0000 7f29d27fc640 10 monclient: dump: 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.054+0000 7f29d959e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:05.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.054+0000 7f29d959e640 1 -- 172.21.15.7:0/3109762236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d40a9d10 msgr2=0x7f29d413f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.054+0000 7f29d959e640 1 --2- 172.21.15.7:0/3109762236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d40a9d10 0x7f29d413f080 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f29d40b54d0 tx=0x7f29bc02b910 comp rx=0 tx=0).stop 2024-01-21T03:20:05.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.054+0000 7f29d959e640 1 -- 172.21.15.7:0/3109762236 shutdown_connections 2024-01-21T03:20:05.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.054+0000 7f29d959e640 1 --2- 172.21.15.7:0/3109762236 >> v2:172.21.15.7:3300/0 conn(0x7f29d40b2310 0x7f29d413f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.054+0000 7f29d959e640 1 --2- 172.21.15.7:0/3109762236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d40a9d10 0x7f29d413f080 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.054+0000 7f29d959e640 1 --2- 172.21.15.7:0/3109762236 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f29d40a8b20 0x7f29d4145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.054+0000 7f29d959e640 1 -- 172.21.15.7:0/3109762236 >> 172.21.15.7:0/3109762236 conn(0x7f29d4066fb0 msgr2=0x7f29d40b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:05.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.055+0000 7f29d959e640 1 -- 172.21.15.7:0/3109762236 shutdown_connections 2024-01-21T03:20:05.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.055+0000 7f29d959e640 1 -- 172.21.15.7:0/3109762236 wait complete. 2024-01-21T03:20:05.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.056+0000 7f29d959e640 10 monclient: build_initial_monmap 2024-01-21T03:20:05.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.056+0000 7f29d959e640 10 monclient: monmap: 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.057+0000 7f29d959e640 1 librados: starting msgr at 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.057+0000 7f29d959e640 1 librados: starting objecter 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.057+0000 7f29d959e640 1 Processor -- start 2024-01-21T03:20:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.057+0000 7f29d959e640 1 -- start start 2024-01-21T03:20:05.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.057+0000 7f29d959e640 1 librados: setting wanted keys 2024-01-21T03:20:05.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.057+0000 7f29d959e640 1 librados: calling monclient init 2024-01-21T03:20:05.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.057+0000 7f29d959e640 10 monclient: init 2024-01-21T03:20:05.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:05.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:20:05.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29d40a8b20 0x7f29d40b5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:05.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 10 monclient(hunting): picked mon.noname-a con 0x7f29d40a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:05.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7fb397a0e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T03:20:05.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7fb397a0e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T03:20:05.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7fb397a0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:05.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d40a9d10 0x7f29d40b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:05.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 10 monclient(hunting): picked mon.noname-b con 0x7f29d40a9d10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:05.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f29d40b2310 0x7f29d41da6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:05.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 10 monclient(hunting): picked mon.noname-c con 0x7f29d40b2310 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:05.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:05.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d40b6d80 con 0x7f29d40a8b20 2024-01-21T03:20:05.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:05.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d40b6780 con 0x7f29d40a9d10 2024-01-21T03:20:05.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:05.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d41430b0 con 0x7f29d40b2310 2024-01-21T03:20:05.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:05.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7f29d959e640 10 monclient(hunting): authenticate will time out at 1351.374393s 2024-01-21T03:20:05.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.058+0000 7fb397a0e640 1 -- 172.21.15.7:0/124933700 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb39004e970 con 0x7fb3900b3e40 2024-01-21T03:20:05.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.062+0000 7fabe3fff640 1 -- 172.21.15.7:0/1199614872 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fabe4017090 con 0x7fabf0035240 2024-01-21T03:20:05.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.062+0000 7fabe3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T03:20:05.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.062+0000 7fabe3fff640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:05.049 INFO:teuthology.orchestra.run.smithi007.stdout:77309411336 2024-01-21T03:20:05.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe654ff9640 1 -- 172.21.15.7:0/1517079039 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0035f0 con 0x7fe6580a9d10 2024-01-21T03:20:05.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe654ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:05.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe654ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:05.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe654ff9640 10 monclient: dump: 2024-01-21T03:20:05.051 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:05.051 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:05.051 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:05.051 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:05.051 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:05.051 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:05.051 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:05.051 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe654ff9640 10 monclient: _finish_auth 0 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe654ff9640 10 monclient: _check_auth_tickets 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe654ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe654ff9640 1 -- 172.21.15.7:0/1517079039 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64c003770 con 0x7fe6580a9d10 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe654ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe654ff9640 1 -- 172.21.15.7:0/1517079039 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c003b60 con 0x7fe6580a9d10 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe654ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe654ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe654ff9640 10 monclient: dump: 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe65e8c2640 5 monclient: authenticate success, global_id 4342 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe65e8c2640 10 monclient: _renew_subs 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe65e8c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe65e8c2640 1 -- 172.21.15.7:0/1517079039 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65813a490 con 0x7fe6580a9d10 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe6567fc640 4 set_mon_vals no callback set 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe65e8c2640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe65e8c2640 10 monclient: _renew_subs 2024-01-21T03:20:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe65e8c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe65e8c2640 1 -- 172.21.15.7:0/1517079039 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe658131830 con 0x7fe6580a9d10 2024-01-21T03:20:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.063+0000 7fe65e8c2640 1 librados: init done 2024-01-21T03:20:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.064+0000 7fe65e8c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.064+0000 7fe65e8c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.064+0000 7fe65e8c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.064+0000 7fe65e8c2640 1 -- 172.21.15.7:0/1517079039 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65813b890 con 0x7fe6580a9d10 2024-01-21T03:20:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.064+0000 7fabe1ffb640 10 librados: watch_flush enter 2024-01-21T03:20:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.064+0000 7fabe1ffb640 10 librados: watch_flush exit 2024-01-21T03:20:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.064+0000 7fabe1ffb640 20 client.4330.objecter shutdown clearing up homeless session... 2024-01-21T03:20:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.064+0000 7fabe1ffb640 10 client.4330.objecter successfully canceled tick 2024-01-21T03:20:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.064+0000 7fabe1ffb640 1 -- 172.21.15.7:0/1199614872 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fabdc06c8e0 msgr2=0x7fabdc06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 1 --2- 172.21.15.7:0/1199614872 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fabdc06c8e0 0x7fabdc06eda0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fabec035000 tx=0x7fabec030000 comp rx=0 tx=0).stop 2024-01-21T03:20:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 10 monclient: shutdown 2024-01-21T03:20:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 1 -- 172.21.15.7:0/1199614872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf0035240 msgr2=0x7fabf00b8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 1 --2- 172.21.15.7:0/1199614872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf0035240 0x7fabf00b8fe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fabe400e7e0 tx=0x7fabe400eca0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 1 -- 172.21.15.7:0/1199614872 shutdown_connections 2024-01-21T03:20:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 1 --2- 172.21.15.7:0/1199614872 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fabdc06c8e0 0x7fabdc06eda0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 1 --2- 172.21.15.7:0/1199614872 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabf00bd9f0 0x7fabf00c1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 1 --2- 172.21.15.7:0/1199614872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf0035240 0x7fabf00b8fe0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 1 --2- 172.21.15.7:0/1199614872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf00326d0 0x7fabf00c47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 1 -- 172.21.15.7:0/1199614872 >> 172.21.15.7:0/1199614872 conn(0x7fabf00060c0 msgr2=0x7fabf00347d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 1 -- 172.21.15.7:0/1199614872 shutdown_connections 2024-01-21T03:20:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 1 -- 172.21.15.7:0/1199614872 wait complete. 2024-01-21T03:20:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7fabe1ffb640 1 librados: shutdown 2024-01-21T03:20:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.065+0000 7f29cbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d40a9d10 0x7f29d40b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:05.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.066+0000 7f1689d3c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T03:20:05.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.066+0000 7f1689d3c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T03:20:05.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.066+0000 7f1689d3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:05.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.066+0000 7f1689d3c640 1 -- 172.21.15.7:0/2317196430 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f168404e970 con 0x7f168406f670 2024-01-21T03:20:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.069+0000 7f29cbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d40a9d10 0x7f29d40b6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36796/0 (socket says 172.21.15.7:36796) 2024-01-21T03:20:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.069+0000 7f29cbfff640 1 -- 172.21.15.7:0/34168596 learned_addr learned my addr 172.21.15.7:0/34168596 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.069+0000 7f29cbfff640 10 monclient(hunting): get_auth_request con 0x7f29d40a9d10 auth_method 0 2024-01-21T03:20:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.069+0000 7f29cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.069+0000 7f29cbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.069+0000 7f29cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:05.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.071+0000 7f29cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:05.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.071+0000 7f29cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:05.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.071+0000 7f29cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:05.066 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411337 got 77309411336 for osd.6 2024-01-21T03:20:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.133+0000 7fe2c5ffb640 1 -- 172.21.15.7:0/2944573048 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe2cc003ca0 con 0x7fe2d00a8b20 2024-01-21T03:20:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.133+0000 7fe2c5ffb640 1 --2- 172.21.15.7:0/2944573048 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fe2b406c860 0x7fe2b406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.133+0000 7fe2d4a5d640 1 --2- 172.21.15.7:0/2944573048 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fe2b406c860 0x7fe2b406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.133+0000 7fe2d4a5d640 10 monclient: get_auth_request con 0x7fe2b406c860 auth_method 0 2024-01-21T03:20:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.134+0000 7fe2c5ffb640 10 client.4295.objecter ms_handle_connect 0x7fe2b406c860 2024-01-21T03:20:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.134+0000 7fe2d4a5d640 1 --2- 172.21.15.7:0/2944573048 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fe2b406c860 0x7fe2b406ed20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0132aa0 tx=0x7fe2c0006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:05.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.146+0000 7fe2c5ffb640 1 -- 172.21.15.7:0/2944573048 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe2cc058250 con 0x7fe2d00a8b20 2024-01-21T03:20:05.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.146+0000 7fe2c5ffb640 10 client.4295.objecter ms_dispatch 0x7fe2d0000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:05.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.146+0000 7fe2c5ffb640 3 client.4295.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:05.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.146+0000 7fe2c5ffb640 3 client.4295.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:05.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.146+0000 7fe2c5ffb640 20 client.4295.objecter dump_active .. 0 homeless 2024-01-21T03:20:05.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.147+0000 7fe2c5ffb640 1 -- 172.21.15.7:0/2944573048 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe2cc053e90 con 0x7fe2d00a8b20 2024-01-21T03:20:05.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.147+0000 7fe2c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:05.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.147+0000 7fe2c5ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.152+0000 7f29d37fe640 1 --2- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f29d40b2310 0x7f29d41da6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.156+0000 7f29d37fe640 10 monclient(hunting): get_auth_request con 0x7f29d40b2310 auth_method 0 2024-01-21T03:20:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.156+0000 7f29d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.156+0000 7f29d37fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.156+0000 7f29d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:05.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.161+0000 7f29d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:05.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.161+0000 7f29d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:05.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.161+0000 7f29d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:05.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.168+0000 7f29d37fe640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2024-01-21T03:20:05.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.168+0000 7f29d37fe640 1 -- 172.21.15.7:0/34168596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d40a9d10 msgr2=0x7f29d40b6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.168+0000 7f29d37fe640 1 --2- 172.21.15.7:0/34168596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d40a9d10 0x7f29d40b6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.168+0000 7f29d37fe640 1 -- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29d40a8b20 msgr2=0x7f29d40b5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.168+0000 7f29d37fe640 1 --2- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29d40a8b20 0x7f29d40b5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.168+0000 7f29d37fe640 10 monclient: _finish_hunting 0 2024-01-21T03:20:05.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.168+0000 7f29d37fe640 1 monclient: found mon.noname-c 2024-01-21T03:20:05.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.168+0000 7f29d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:05.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.168+0000 7f29d37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:20:05.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.168+0000 7f29d37fe640 1 -- 172.21.15.7:0/34168596 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc0094d0 con 0x7f29d40b2310 2024-01-21T03:20:05.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.171+0000 7f29d37fe640 1 --2- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f29d40b2310 0x7f29d41da6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00bdf0 tx=0x7f29cc00bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:05.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.171+0000 7f29cb7fe640 10 client.?.objecter ms_handle_connect 0x7f29d40b2310 2024-01-21T03:20:05.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.171+0000 7f29cb7fe640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:05.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.172+0000 7fb385ffb640 1 -- 172.21.15.7:0/124933700 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb38c01e090 con 0x7fb3900b3e40 2024-01-21T03:20:05.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.172+0000 7fb385ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T03:20:05.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.172+0000 7fb385ffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:05.158 INFO:teuthology.orchestra.run.smithi007.stdout:90194313224 2024-01-21T03:20:05.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 10 librados: watch_flush enter 2024-01-21T03:20:05.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 10 librados: watch_flush exit 2024-01-21T03:20:05.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 20 client.4292.objecter shutdown clearing up homeless session... 2024-01-21T03:20:05.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 10 client.4292.objecter successfully canceled tick 2024-01-21T03:20:05.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 1 -- 172.21.15.7:0/124933700 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb37406c8b0 msgr2=0x7fb37406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 1 --2- 172.21.15.7:0/124933700 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb37406c8b0 0x7fb37406ed70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb388035000 tx=0x7fb38802e000 comp rx=0 tx=0).stop 2024-01-21T03:20:05.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 10 monclient: shutdown 2024-01-21T03:20:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 1 -- 172.21.15.7:0/124933700 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 msgr2=0x7fb390136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 1 --2- 172.21.15.7:0/124933700 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 0x7fb390136f80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00b4a0 tx=0x7fb38c00b960 comp rx=0 tx=0).stop 2024-01-21T03:20:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 1 -- 172.21.15.7:0/124933700 shutdown_connections 2024-01-21T03:20:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 1 --2- 172.21.15.7:0/124933700 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb37406c8b0 0x7fb37406ed70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 1 --2- 172.21.15.7:0/124933700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3901417e0 0x7fb3901374c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 1 --2- 172.21.15.7:0/124933700 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb3900b3e40 0x7fb390136f80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 1 --2- 172.21.15.7:0/124933700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb39006e990 0x7fb39013daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 1 -- 172.21.15.7:0/124933700 >> 172.21.15.7:0/124933700 conn(0x7fb39006b270 msgr2=0x7fb3900b08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 1 -- 172.21.15.7:0/124933700 shutdown_connections 2024-01-21T03:20:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 1 -- 172.21.15.7:0/124933700 wait complete. 2024-01-21T03:20:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.175+0000 7fb397a0e640 1 librados: shutdown 2024-01-21T03:20:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29cb7fe640 1 -- 172.21.15.7:0/34168596 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc00ce70 con 0x7f29d40b2310 2024-01-21T03:20:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29cb7fe640 10 monclient: dump: 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29cb7fe640 10 monclient: _finish_auth 0 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29cb7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:20:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29cb7fe640 1 -- 172.21.15.7:0/34168596 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29cc014440 con 0x7f29d40b2310 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29cb7fe640 1 -- 172.21.15.7:0/34168596 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc014810 con 0x7f29d40b2310 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29d1ffb640 4 set_mon_vals no callback set 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29cb7fe640 10 monclient: dump: 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29d959e640 5 monclient: authenticate success, global_id 4298 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29d959e640 10 monclient: _renew_subs 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29d959e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29d959e640 1 -- 172.21.15.7:0/34168596 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d41433a0 con 0x7f29d40b2310 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29d959e640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29d959e640 10 monclient: _renew_subs 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29d959e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29d959e640 1 -- 172.21.15.7:0/34168596 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d41daea0 con 0x7f29d40b2310 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.177+0000 7f29d959e640 1 librados: init done 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.178+0000 7f29d959e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.178+0000 7f29d959e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.178+0000 7f29d959e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.178+0000 7f29d959e640 1 -- 172.21.15.7:0/34168596 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d404e970 con 0x7f29d40b2310 2024-01-21T03:20:05.172 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313224 for osd.0 2024-01-21T03:20:05.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.255+0000 7ff23d061640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T03:20:05.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.255+0000 7ff23d061640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T03:20:05.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.255+0000 7ff23d061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:05.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.255+0000 7ff23d061640 1 -- 172.21.15.7:0/813608811 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff230033300 con 0x7ff2300c0840 2024-01-21T03:20:05.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.274+0000 7f09d2ffd640 1 -- 172.21.15.7:0/2635695433 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f09d40071d0 con 0x7f09dc13a8c0 2024-01-21T03:20:05.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.274+0000 7f09d2ffd640 1 --2- 172.21.15.7:0/2635695433 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f09c006c890 0x7f09c006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:05.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.274+0000 7f09e2065640 1 --2- 172.21.15.7:0/2635695433 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f09c006c890 0x7f09c006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:05.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.275+0000 7f09e2065640 10 monclient: get_auth_request con 0x7f09c006c890 auth_method 0 2024-01-21T03:20:05.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.275+0000 7f09e2065640 1 --2- 172.21.15.7:0/2635695433 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f09c006c890 0x7f09c006ed50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f09d8034000 tx=0x7f09d802f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:05.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.275+0000 7f09d2ffd640 10 client.4302.objecter ms_handle_connect 0x7f09c006c890 2024-01-21T03:20:05.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.289+0000 7f09d2ffd640 1 -- 172.21.15.7:0/2635695433 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f09d4059440 con 0x7f09dc13a8c0 2024-01-21T03:20:05.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.289+0000 7f09d2ffd640 10 client.4302.objecter ms_dispatch 0x7f09dc000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:05.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.289+0000 7f09d2ffd640 3 client.4302.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:05.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.289+0000 7f09d2ffd640 3 client.4302.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:05.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.289+0000 7f09d2ffd640 20 client.4302.objecter dump_active .. 0 homeless 2024-01-21T03:20:05.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.289+0000 7f09d2ffd640 1 -- 172.21.15.7:0/2635695433 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f09d4055080 con 0x7f09dc13a8c0 2024-01-21T03:20:05.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.289+0000 7f09d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:05.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.289+0000 7f09d2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:05.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.305+0000 7f16817fa640 1 -- 172.21.15.7:0/2317196430 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1674021090 con 0x7f168406f670 2024-01-21T03:20:05.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.305+0000 7f16817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T03:20:05.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.305+0000 7f16817fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:05.292 INFO:teuthology.orchestra.run.smithi007.stdout:60129542151 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 10 librados: watch_flush enter 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 10 librados: watch_flush exit 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 20 client.4299.objecter shutdown clearing up homeless session... 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 10 client.4299.objecter successfully canceled tick 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 1 -- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f167006c8b0 msgr2=0x7f167006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f167006c8b0 0x7f167006ed70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f167c02bf20 tx=0x7f167c030000 comp rx=0 tx=0).stop 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 10 monclient: shutdown 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 1 -- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f168406f670 msgr2=0x7f16840abf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f168406f670 0x7f16840abf10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f167400e850 tx=0x7f167400ed10 comp rx=0 tx=0).stop 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 1 -- 172.21.15.7:0/2317196430 shutdown_connections 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f167006c8b0 0x7f167006ed70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1684146200 0x7f16840ac450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f168406f670 0x7f16840abf10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 1 --2- 172.21.15.7:0/2317196430 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f168406f290 0x7f16841459b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 1 -- 172.21.15.7:0/2317196430 >> 172.21.15.7:0/2317196430 conn(0x7f16840b0040 msgr2=0x7f1684142ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:05.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 1 -- 172.21.15.7:0/2317196430 shutdown_connections 2024-01-21T03:20:05.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.308+0000 7f1689d3c640 1 -- 172.21.15.7:0/2317196430 wait complete. 2024-01-21T03:20:05.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.309+0000 7f1689d3c640 1 librados: shutdown 2024-01-21T03:20:05.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.314+0000 7fe654ff9640 1 -- 172.21.15.7:0/1517079039 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe64c003cc0 con 0x7fe6580a9d10 2024-01-21T03:20:05.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.314+0000 7fe654ff9640 1 --2- 172.21.15.7:0/1517079039 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fe64406c8b0 0x7fe64406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:05.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.314+0000 7fe657fff640 1 --2- 172.21.15.7:0/1517079039 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fe64406c8b0 0x7fe64406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:05.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.314+0000 7fe657fff640 10 monclient: get_auth_request con 0x7fe64406c8b0 auth_method 0 2024-01-21T03:20:05.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.315+0000 7fe657fff640 1 --2- 172.21.15.7:0/1517079039 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fe64406c8b0 0x7fe64406ed70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe648006fd0 tx=0x7fe648006bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:05.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.315+0000 7fe654ff9640 10 client.4342.objecter ms_handle_connect 0x7fe64406c8b0 2024-01-21T03:20:05.308 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.4 2024-01-21T03:20:05.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.323+0000 7fe654ff9640 1 -- 172.21.15.7:0/1517079039 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe64c0010e0 con 0x7fe6580a9d10 2024-01-21T03:20:05.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.323+0000 7fe654ff9640 10 client.4342.objecter ms_dispatch 0x7fe658000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:05.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.323+0000 7fe654ff9640 3 client.4342.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:05.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.323+0000 7fe654ff9640 3 client.4342.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:05.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.323+0000 7fe654ff9640 20 client.4342.objecter dump_active .. 0 homeless 2024-01-21T03:20:05.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.333+0000 7fe654ff9640 1 -- 172.21.15.7:0/1517079039 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe64c0541b0 con 0x7fe6580a9d10 2024-01-21T03:20:05.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.333+0000 7fe654ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:05.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.333+0000 7fe654ff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:05.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.340+0000 7ff21f7fe640 1 -- 172.21.15.7:0/813608811 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff2380d1f40 con 0x7ff2300c0840 2024-01-21T03:20:05.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.340+0000 7ff21f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T03:20:05.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.340+0000 7ff21f7fe640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:05.326 INFO:teuthology.orchestra.run.smithi007.stdout:94489280516 2024-01-21T03:20:05.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.343+0000 7ff23d061640 10 librados: watch_flush enter 2024-01-21T03:20:05.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.343+0000 7ff23d061640 10 librados: watch_flush exit 2024-01-21T03:20:05.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.343+0000 7ff23d061640 20 client.4339.objecter shutdown clearing up homeless session... 2024-01-21T03:20:05.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.343+0000 7ff23d061640 10 client.4339.objecter successfully canceled tick 2024-01-21T03:20:05.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.343+0000 7ff23d061640 1 -- 172.21.15.7:0/813608811 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ff22406c890 msgr2=0x7ff22406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.343+0000 7ff23d061640 1 --2- 172.21.15.7:0/813608811 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ff22406c890 0x7ff22406ed50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff228005ae0 tx=0x7ff228001d50 comp rx=0 tx=0).stop 2024-01-21T03:20:05.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.343+0000 7ff23d061640 10 monclient: shutdown 2024-01-21T03:20:05.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.343+0000 7ff23d061640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:05.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.343+0000 7ff23d061640 1 -- 172.21.15.7:0/813608811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2300c0840 msgr2=0x7ff2300be640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.343+0000 7ff23d061640 1 --2- 172.21.15.7:0/813608811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2300c0840 0x7ff2300be640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff238050150 tx=0x7ff238067a90 comp rx=0 tx=0).stop 2024-01-21T03:20:05.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.344+0000 7ff23d061640 1 -- 172.21.15.7:0/813608811 shutdown_connections 2024-01-21T03:20:05.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.344+0000 7ff23d061640 1 --2- 172.21.15.7:0/813608811 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ff22406c890 0x7ff22406ed50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.344+0000 7ff23d061640 1 --2- 172.21.15.7:0/813608811 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2300c20b0 0x7ff2300beb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.344+0000 7ff23d061640 1 --2- 172.21.15.7:0/813608811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2300c0840 0x7ff2300be640 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff238050150 tx=0x7ff238067a90 comp rx=0 tx=0).stop 2024-01-21T03:20:05.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.344+0000 7ff23d061640 1 --2- 172.21.15.7:0/813608811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2200052a0 0x7ff2300c0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.344+0000 7ff23d061640 1 -- 172.21.15.7:0/813608811 >> 172.21.15.7:0/813608811 conn(0x7ff230006610 msgr2=0x7ff23003bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:05.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.344+0000 7ff23d061640 1 -- 172.21.15.7:0/813608811 shutdown_connections 2024-01-21T03:20:05.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.344+0000 7ff23d061640 1 -- 172.21.15.7:0/813608811 wait complete. 2024-01-21T03:20:05.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.344+0000 7ff23d061640 1 librados: shutdown 2024-01-21T03:20:05.342 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.3 2024-01-21T03:20:05.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.458+0000 7fe2d74c2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T03:20:05.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.458+0000 7fe2d74c2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T03:20:05.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.458+0000 7fe2d74c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:05.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.458+0000 7fe2d74c2640 1 -- 172.21.15.7:0/2944573048 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe2d01408e0 con 0x7fe2d00a8b20 2024-01-21T03:20:05.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.526+0000 7f29cb7fe640 1 -- 172.21.15.7:0/34168596 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f29cc01d3b0 con 0x7f29d40b2310 2024-01-21T03:20:05.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.527+0000 7f29cb7fe640 1 --2- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f29b406c860 0x7f29b406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:05.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.527+0000 7f29d2ffd640 1 --2- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f29b406c860 0x7f29b406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:05.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.527+0000 7f29d2ffd640 10 monclient: get_auth_request con 0x7f29b406c860 auth_method 0 2024-01-21T03:20:05.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.527+0000 7f29cb7fe640 10 client.4298.objecter ms_handle_connect 0x7f29b406c860 2024-01-21T03:20:05.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.527+0000 7f29d2ffd640 1 --2- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f29b406c860 0x7f29b406ed20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009710 tx=0x7f29c4006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:05.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.538+0000 7f09e42c9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T03:20:05.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.538+0000 7f09e42c9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T03:20:05.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.538+0000 7f09e42c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:05.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.538+0000 7f09e42c9640 1 -- 172.21.15.7:0/2635695433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f09dc1db820 con 0x7f09dc13a8c0 2024-01-21T03:20:05.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.540+0000 7f29cb7fe640 1 -- 172.21.15.7:0/34168596 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f29cc0587e0 con 0x7f29d40b2310 2024-01-21T03:20:05.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.540+0000 7f29cb7fe640 10 client.4298.objecter ms_dispatch 0x7f29d4000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:05.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.540+0000 7f29cb7fe640 3 client.4298.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:05.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.540+0000 7f29cb7fe640 3 client.4298.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:05.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.540+0000 7f29cb7fe640 20 client.4298.objecter dump_active .. 0 homeless 2024-01-21T03:20:05.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.540+0000 7f29cb7fe640 1 -- 172.21.15.7:0/34168596 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f29cc054b20 con 0x7f29d40b2310 2024-01-21T03:20:05.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.540+0000 7f29cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:05.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.540+0000 7f29cb7fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:05.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.546+0000 7f09d2ffd640 1 -- 172.21.15.7:0/2635695433 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f09d40586a0 con 0x7f09dc13a8c0 2024-01-21T03:20:05.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.546+0000 7f09d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T03:20:05.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.546+0000 7f09d2ffd640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:05.532 INFO:teuthology.orchestra.run.smithi007.stdout:77309411337 2024-01-21T03:20:05.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.548+0000 7f09e42c9640 10 librados: watch_flush enter 2024-01-21T03:20:05.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.548+0000 7f09e42c9640 10 librados: watch_flush exit 2024-01-21T03:20:05.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 20 client.4302.objecter shutdown clearing up homeless session... 2024-01-21T03:20:05.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 10 client.4302.objecter successfully canceled tick 2024-01-21T03:20:05.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 1 -- 172.21.15.7:0/2635695433 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f09c006c890 msgr2=0x7f09c006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 1 --2- 172.21.15.7:0/2635695433 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f09c006c890 0x7f09c006ed50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f09d8034000 tx=0x7f09d802f000 comp rx=0 tx=0).stop 2024-01-21T03:20:05.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 10 monclient: shutdown 2024-01-21T03:20:05.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:05.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 1 -- 172.21.15.7:0/2635695433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc13a8c0 msgr2=0x7f09dc0b0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 1 --2- 172.21.15.7:0/2635695433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc13a8c0 0x7f09dc0b0d50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0b41b0 tx=0x7f09d400b4a0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 1 -- 172.21.15.7:0/2635695433 shutdown_connections 2024-01-21T03:20:05.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 1 --2- 172.21.15.7:0/2635695433 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f09c006c890 0x7f09c006ed50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 1 --2- 172.21.15.7:0/2635695433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc13a8c0 0x7f09dc0b0d50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 1 --2- 172.21.15.7:0/2635695433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09dc1db150 0x7f09dc0b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 1 --2- 172.21.15.7:0/2635695433 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09c80039c0 0x7f09dc1dac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 1 -- 172.21.15.7:0/2635695433 >> 172.21.15.7:0/2635695433 conn(0x7f09dc066fb0 msgr2=0x7f09dc13cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:05.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 1 -- 172.21.15.7:0/2635695433 shutdown_connections 2024-01-21T03:20:05.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 1 -- 172.21.15.7:0/2635695433 wait complete. 2024-01-21T03:20:05.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.549+0000 7f09e42c9640 1 librados: shutdown 2024-01-21T03:20:05.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.552+0000 7fe2c5ffb640 1 -- 172.21.15.7:0/2944573048 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe2d01408e0 con 0x7fe2d00a8b20 2024-01-21T03:20:05.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.552+0000 7fe2c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T03:20:05.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.552+0000 7fe2c5ffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:05.538 INFO:teuthology.orchestra.run.smithi007.stdout:90194313223 2024-01-21T03:20:05.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.555+0000 7fe2d74c2640 10 librados: watch_flush enter 2024-01-21T03:20:05.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.555+0000 7fe2d74c2640 10 librados: watch_flush exit 2024-01-21T03:20:05.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.555+0000 7fe2d74c2640 20 client.4295.objecter shutdown clearing up homeless session... 2024-01-21T03:20:05.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.555+0000 7fe2d74c2640 10 client.4295.objecter successfully canceled tick 2024-01-21T03:20:05.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.555+0000 7fe2d74c2640 1 -- 172.21.15.7:0/2944573048 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fe2b406c860 msgr2=0x7fe2b406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.555+0000 7fe2d74c2640 1 --2- 172.21.15.7:0/2944573048 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fe2b406c860 0x7fe2b406ed20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0132aa0 tx=0x7fe2c0006c90 comp rx=0 tx=0).stop 2024-01-21T03:20:05.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.555+0000 7fe2d74c2640 10 monclient: shutdown 2024-01-21T03:20:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.555+0000 7fe2d74c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.555+0000 7fe2d74c2640 1 -- 172.21.15.7:0/2944573048 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00a8b20 msgr2=0x7fe2d0133e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.555+0000 7fe2d74c2640 1 --2- 172.21.15.7:0/2944573048 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00a8b20 0x7fe2d0133e20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00c410 tx=0x7fe2cc00c8d0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.556+0000 7fe2d74c2640 1 -- 172.21.15.7:0/2944573048 shutdown_connections 2024-01-21T03:20:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.556+0000 7fe2d74c2640 1 --2- 172.21.15.7:0/2944573048 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fe2b406c860 0x7fe2b406ed20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.556+0000 7fe2d74c2640 1 --2- 172.21.15.7:0/2944573048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2d00b2310 0x7fe2d01d7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.556+0000 7fe2d74c2640 1 --2- 172.21.15.7:0/2944573048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2d00a9d10 0x7fe2d0133460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.556+0000 7fe2d74c2640 1 --2- 172.21.15.7:0/2944573048 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe2d00a8b20 0x7fe2d0133e20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.556+0000 7fe2d74c2640 1 -- 172.21.15.7:0/2944573048 >> 172.21.15.7:0/2944573048 conn(0x7fe2d0066fb0 msgr2=0x7fe2d013ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.556+0000 7fe2d74c2640 1 -- 172.21.15.7:0/2944573048 shutdown_connections 2024-01-21T03:20:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.556+0000 7fe2d74c2640 1 -- 172.21.15.7:0/2944573048 wait complete. 2024-01-21T03:20:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.556+0000 7fe2d74c2640 1 librados: shutdown 2024-01-21T03:20:05.549 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411338 got 77309411337 for osd.5 2024-01-21T03:20:05.556 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313224 got 90194313223 for osd.1 2024-01-21T03:20:05.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.634+0000 7fe65e8c2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-21T03:20:05.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.634+0000 7fe65e8c2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-21T03:20:05.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.634+0000 7fe65e8c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:05.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.634+0000 7fe65e8c2640 1 -- 172.21.15.7:0/1517079039 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fe65813b890 con 0x7fe6580a9d10 2024-01-21T03:20:05.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.638+0000 7fe654ff9640 1 -- 172.21.15.7:0/1517079039 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe64c021020 con 0x7fe6580a9d10 2024-01-21T03:20:05.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.638+0000 7fe654ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-21T03:20:05.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.638+0000 7fe654ff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:05.624 INFO:teuthology.orchestra.run.smithi007.stdout:60129542151 2024-01-21T03:20:05.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.639+0000 7fe65e8c2640 10 librados: watch_flush enter 2024-01-21T03:20:05.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.639+0000 7fe65e8c2640 10 librados: watch_flush exit 2024-01-21T03:20:05.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.639+0000 7fe65e8c2640 20 client.4342.objecter shutdown clearing up homeless session... 2024-01-21T03:20:05.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.639+0000 7fe65e8c2640 10 client.4342.objecter successfully canceled tick 2024-01-21T03:20:05.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 1 -- 172.21.15.7:0/1517079039 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fe64406c8b0 msgr2=0x7fe64406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 1 --2- 172.21.15.7:0/1517079039 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fe64406c8b0 0x7fe64406ed70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe648006fd0 tx=0x7fe648006bc0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 10 monclient: shutdown 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 1 -- 172.21.15.7:0/1517079039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6580a9d10 msgr2=0x7fe65813bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 1 --2- 172.21.15.7:0/1517079039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6580a9d10 0x7fe65813bb70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00c400 tx=0x7fe64c00c8c0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 1 -- 172.21.15.7:0/1517079039 shutdown_connections 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 1 --2- 172.21.15.7:0/1517079039 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fe64406c8b0 0x7fe64406ed70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 1 --2- 172.21.15.7:0/1517079039 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6580b2310 0x7fe658141fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 1 --2- 172.21.15.7:0/1517079039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6580a9d10 0x7fe65813bb70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 1 --2- 172.21.15.7:0/1517079039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6580a8b20 0x7fe65813ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 1 -- 172.21.15.7:0/1517079039 >> 172.21.15.7:0/1517079039 conn(0x7fe658066fb0 msgr2=0x7fe6580b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 1 -- 172.21.15.7:0/1517079039 shutdown_connections 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 1 -- 172.21.15.7:0/1517079039 wait complete. 2024-01-21T03:20:05.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.640+0000 7fe65e8c2640 1 librados: shutdown 2024-01-21T03:20:05.636 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.7 2024-01-21T03:20:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.745+0000 7f29d959e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-21T03:20:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.745+0000 7f29d959e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-21T03:20:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.745+0000 7f29d959e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.745+0000 7f29d959e640 1 -- 172.21.15.7:0/34168596 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f29d4072860 con 0x7f29d40b2310 2024-01-21T03:20:05.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.749+0000 7f29cb7fe640 1 -- 172.21.15.7:0/34168596 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f29cc019070 con 0x7f29d40b2310 2024-01-21T03:20:05.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.749+0000 7f29cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-21T03:20:05.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.749+0000 7f29cb7fe640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:05.735 INFO:teuthology.orchestra.run.smithi007.stdout:90194313223 2024-01-21T03:20:05.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 10 librados: watch_flush enter 2024-01-21T03:20:05.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 10 librados: watch_flush exit 2024-01-21T03:20:05.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 20 client.4298.objecter shutdown clearing up homeless session... 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 10 client.4298.objecter successfully canceled tick 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 1 -- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f29b406c860 msgr2=0x7f29b406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 1 --2- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f29b406c860 0x7f29b406ed20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009710 tx=0x7f29c4006c90 comp rx=0 tx=0).stop 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 10 monclient: shutdown 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 1 -- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f29d40b2310 msgr2=0x7f29d41da6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 1 --2- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f29d40b2310 0x7f29d41da6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00bdf0 tx=0x7f29cc00bee0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 1 -- 172.21.15.7:0/34168596 shutdown_connections 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 1 --2- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f29b406c860 0x7f29b406ed20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 1 --2- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f29d40b2310 0x7f29d41da6c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 1 --2- 172.21.15.7:0/34168596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d40a9d10 0x7f29d40b6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 1 --2- 172.21.15.7:0/34168596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29d40a8b20 0x7f29d40b5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 1 -- 172.21.15.7:0/34168596 >> 172.21.15.7:0/34168596 conn(0x7f29d4066fb0 msgr2=0x7f29d413ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:05.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 1 -- 172.21.15.7:0/34168596 shutdown_connections 2024-01-21T03:20:05.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 1 -- 172.21.15.7:0/34168596 wait complete. 2024-01-21T03:20:05.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:05.751+0000 7f29d959e640 1 librados: shutdown 2024-01-21T03:20:05.748 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313224 got 90194313223 for osd.2 2024-01-21T03:20:06.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-21T03:20:06.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.183+0000 7f9838947640 10 monclient: get_monmap_and_config 2024-01-21T03:20:06.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.183+0000 7f9838947640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.183+0000 7f9838947640 10 monclient: monmap: 2024-01-21T03:20:06.169 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.169 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.169 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:06.184645+0000 2024-01-21T03:20:06.169 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:06.184645+0000 2024-01-21T03:20:06.170 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.170 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.170 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:06.170 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:06.170 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:06.170 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:06.170 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:06.170 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.184+0000 7f9838947640 1 Processor -- start 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.184+0000 7f9838947640 1 -- start start 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.184+0000 7f9838947640 10 monclient: init 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.184+0000 7f9838947640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.184+0000 7f9838947640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.184+0000 7f9838947640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98340a8ab0 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.184+0000 7f9838947640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98340a8e90 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.184+0000 7f9838947640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f983413f950 0x7f983414d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.184+0000 7f9838947640 10 monclient(hunting): picked mon.noname-a con 0x7f983413f950 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.184+0000 7f9838947640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9834142dc0 con 0x7f98340a8ab0 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.184+0000 7f9838947640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98340aaa20 con 0x7f98340a8e90 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.184+0000 7f9838947640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.185+0000 7f9838947640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98340aa820 con 0x7f983413f950 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.185+0000 7f9838947640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.185+0000 7f9838947640 10 monclient(hunting): authenticate will time out at 1352.500757s 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.185+0000 7f9832d9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f983413f950 0x7f983414d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.185+0000 7f982bfff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f98340a8e90 0x7f983413f240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:44442/0 (socket says 172.21.15.7:44442) 2024-01-21T03:20:06.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.185+0000 7f982bfff640 1 -- 172.21.15.7:0/1859392332 learned_addr learned my addr 172.21.15.7:0/1859392332 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:06.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.186+0000 7f9832d9d640 10 monclient(hunting): get_auth_request con 0x7f983413f950 auth_method 0 2024-01-21T03:20:06.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.186+0000 7f9832d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.186+0000 7f9832d9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.186+0000 7f9832d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-21T03:20:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.187+0000 7f9832d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.187+0000 7f9832d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.187+0000 7f9832d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.187+0000 7f9831d9b640 1 -- 172.21.15.7:0/1859392332 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790400935 0 0) 0x7f98340aaa20 con 0x7f98340a8e90 2024-01-21T03:20:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.187+0000 7f9831d9b640 10 monclient(hunting): _negotiate 2024-01-21T03:20:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.187+0000 7f9831d9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.187+0000 7f9831d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.187+0000 7f9831d9b640 10 monclient(hunting): my global_id is 4351 2024-01-21T03:20:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.187+0000 7f9831d9b640 1 -- 172.21.15.7:0/1859392332 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9820003520 con 0x7f98340a8e90 2024-01-21T03:20:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.189+0000 7f9832d9d640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2024-01-21T03:20:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.189+0000 7f9832d9d640 1 -- 172.21.15.7:0/1859392332 >> v1:172.21.15.177:6789/0 conn(0x7f98340a8e90 legacy=0x7f983413f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.189+0000 7f9832d9d640 1 -- 172.21.15.7:0/1859392332 >> v1:172.21.15.7:6789/0 conn(0x7f98340a8ab0 legacy=0x7f98341426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.189+0000 7f9832d9d640 10 monclient: _finish_hunting 0 2024-01-21T03:20:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.189+0000 7f9832d9d640 1 monclient: found mon.noname-a 2024-01-21T03:20:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.189+0000 7f9832d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.189+0000 7f9832d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.189+0000 7f9832d9d640 1 -- 172.21.15.7:0/1859392332 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98340aad00 con 0x7f983413f950 2024-01-21T03:20:06.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.194+0000 7f9832d9d640 1 --2- 172.21.15.7:0/1859392332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f983413f950 0x7f983414d910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f982c009d90 tx=0x7f982c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46628434c25a8c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.198+0000 7f9831d9b640 1 -- 172.21.15.7:0/1859392332 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c02cd00 con 0x7f983413f950 2024-01-21T03:20:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.198+0000 7f9831d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.198+0000 7f9831d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.198+0000 7f9831d9b640 10 monclient: dump: 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.198+0000 7f9831d9b640 10 monclient: _finish_auth 0 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.198+0000 7f9831d9b640 10 monclient: _check_auth_tickets 2024-01-21T03:20:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9831d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9831d9b640 1 -- 172.21.15.7:0/1859392332 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982c02ce60 con 0x7f983413f950 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9838947640 5 monclient: authenticate success, global_id 4305 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9838947640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9831d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9838947640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9831d9b640 1 -- 172.21.15.7:0/1859392332 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c034660 con 0x7f983413f950 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9831d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9838947640 4 set_mon_vals no callback set 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9831d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9831d9b640 10 monclient: dump: 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.187 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.187 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.187 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.187 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.187 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9838947640 10 monclient: shutdown 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9838947640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9838947640 1 -- 172.21.15.7:0/1859392332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f983413f950 msgr2=0x7f983414d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9838947640 1 --2- 172.21.15.7:0/1859392332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f983413f950 0x7f983414d910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f982c009d90 tx=0x7f982c02bf50 comp rx=0 tx=0).stop 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9838947640 1 -- 172.21.15.7:0/1859392332 shutdown_connections 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9838947640 1 --2- 172.21.15.7:0/1859392332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f983413f950 0x7f983414d910 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9838947640 1 -- 172.21.15.7:0/1859392332 >> 172.21.15.7:0/1859392332 conn(0x7f9834066f00 msgr2=0x7f98340b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9838947640 1 -- 172.21.15.7:0/1859392332 shutdown_connections 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.199+0000 7f9838947640 1 -- 172.21.15.7:0/1859392332 wait complete. 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.200+0000 7f9838947640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.200+0000 7f9838947640 10 monclient: monmap: 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 1 librados: starting msgr at 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 1 librados: starting objecter 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 1 Processor -- start 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 1 -- start start 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 1 librados: setting wanted keys 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 1 librados: calling monclient init 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 10 monclient: init 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T03:20:06.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98340a8ab0 0x7f98340ab9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 10 monclient(hunting): picked mon.noname-b con 0x7f98340a8ab0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98340a8e90 0x7f98341460c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 10 monclient(hunting): picked mon.noname-a con 0x7f98340a8e90 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f983413f950 0x7f98340abff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 10 monclient(hunting): picked mon.noname-c con 0x7f983413f950 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98340ad0f0 con 0x7f98340a8e90 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98341d6e70 con 0x7f98340a8ab0 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9834143610 con 0x7f983413f950 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.201+0000 7f9838947640 10 monclient(hunting): authenticate will time out at 1352.517524s 2024-01-21T03:20:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f983259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98340a8ab0 0x7f98340ab9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f982bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98340a8e90 0x7f98341460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f9832d9d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f983413f950 0x7f98340abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f983259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98340a8ab0 0x7f98340ab9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:36800/0 (socket says 172.21.15.7:36800) 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f983259c640 1 -- 172.21.15.7:0/1233089965 learned_addr learned my addr 172.21.15.7:0/1233089965 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f983259c640 10 monclient(hunting): get_auth_request con 0x7f98340a8ab0 auth_method 0 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f983259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f983259c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f983259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f9832d9d640 10 monclient(hunting): get_auth_request con 0x7f983413f950 auth_method 0 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f9832d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f9832d9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f9832d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f982bfff640 10 monclient(hunting): get_auth_request con 0x7f98340a8e90 auth_method 0 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f982bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f982bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.202+0000 7f982bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.203+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.203+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.203+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.203+0000 7f9832d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.203+0000 7f9832d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.203+0000 7f9832d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.203+0000 7f982bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.203+0000 7f982bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.203+0000 7f982bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.205+0000 7f983259c640 10 monclient(hunting): handle_auth_done global_id 4354 payload 824 2024-01-21T03:20:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.205+0000 7f983259c640 1 -- 172.21.15.7:0/1233089965 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f983413f950 msgr2=0x7f98340abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.205+0000 7f983259c640 1 --2- 172.21.15.7:0/1233089965 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f983413f950 0x7f98340abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.205+0000 7f983259c640 1 -- 172.21.15.7:0/1233089965 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98340a8e90 msgr2=0x7f98341460c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.205+0000 7f983259c640 1 --2- 172.21.15.7:0/1233089965 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98340a8e90 0x7f98341460c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.205+0000 7f983259c640 10 monclient: _finish_hunting 0 2024-01-21T03:20:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.205+0000 7f983259c640 1 monclient: found mon.noname-b 2024-01-21T03:20:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.205+0000 7f983259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.205+0000 7f983259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.205+0000 7f983259c640 1 -- 172.21.15.7:0/1233089965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c009a70 con 0x7f98340a8ab0 2024-01-21T03:20:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.208+0000 7f983259c640 1 --2- 172.21.15.7:0/1233089965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98340a8ab0 0x7f98340ab9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f982400c400 tx=0x7f982400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.208+0000 7f982affd640 10 client.?.objecter ms_handle_connect 0x7f98340a8ab0 2024-01-21T03:20:06.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.208+0000 7f982affd640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:06.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f982affd640 1 -- 172.21.15.7:0/1233089965 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98240052d0 con 0x7f98340a8ab0 2024-01-21T03:20:06.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f982affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f982affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:06.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f982affd640 10 monclient: dump: 2024-01-21T03:20:06.198 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.198 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.198 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.198 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.198 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f982affd640 10 monclient: _finish_auth 0 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f982affd640 10 monclient: _check_auth_tickets 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f982affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f982affd640 1 -- 172.21.15.7:0/1233089965 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9824005450 con 0x7f98340a8ab0 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f9838947640 5 monclient: authenticate success, global_id 4354 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f9838947640 10 monclient: _renew_subs 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f9838947640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f9838947640 1 -- 172.21.15.7:0/1233089965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9834137be0 con 0x7f98340a8ab0 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f982affd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f982affd640 1 -- 172.21.15.7:0/1233089965 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9824007260 con 0x7f98340a8ab0 2024-01-21T03:20:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f982affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f982affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f982affd640 10 monclient: dump: 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f9830d99640 4 set_mon_vals no callback set 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f9838947640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f9838947640 10 monclient: _renew_subs 2024-01-21T03:20:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f9838947640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f9838947640 1 -- 172.21.15.7:0/1233089965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9834138090 con 0x7f98340a8ab0 2024-01-21T03:20:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.211+0000 7f9838947640 1 librados: init done 2024-01-21T03:20:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.212+0000 7f9838947640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.212+0000 7f9838947640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.212+0000 7f9838947640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.212+0000 7f9838947640 1 -- 172.21.15.7:0/1233089965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983404e970 con 0x7f98340a8ab0 2024-01-21T03:20:06.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.232+0000 7f982affd640 1 -- 172.21.15.7:0/1233089965 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f98240073c0 con 0x7f98340a8ab0 2024-01-21T03:20:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.232+0000 7f982affd640 1 --2- 172.21.15.7:0/1233089965 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f981406c8b0 0x7f981406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.232+0000 7f982bfff640 1 --2- 172.21.15.7:0/1233089965 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f981406c8b0 0x7f981406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.232+0000 7f982bfff640 10 monclient: get_auth_request con 0x7f981406c8b0 auth_method 0 2024-01-21T03:20:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.232+0000 7f982bfff640 1 --2- 172.21.15.7:0/1233089965 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f981406c8b0 0x7f981406ed70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98341db4b0 tx=0x7f981c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.232+0000 7f982affd640 10 client.4354.objecter ms_handle_connect 0x7f981406c8b0 2024-01-21T03:20:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.235+0000 7f982affd640 1 -- 172.21.15.7:0/1233089965 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9824007670 con 0x7f98340a8ab0 2024-01-21T03:20:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.235+0000 7f982affd640 10 client.4354.objecter ms_dispatch 0x7f9834000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.235+0000 7f982affd640 3 client.4354.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.235+0000 7f982affd640 3 client.4354.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.235+0000 7f982affd640 20 client.4354.objecter dump_active .. 0 homeless 2024-01-21T03:20:06.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.292+0000 7f9de1cd7640 10 monclient: get_monmap_and_config 2024-01-21T03:20:06.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.292+0000 7f9de1cd7640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.292+0000 7f9de1cd7640 10 monclient: monmap: 2024-01-21T03:20:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:06.293781+0000 2024-01-21T03:20:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:06.293781+0000 2024-01-21T03:20:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:06.279 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.293+0000 7f9de1cd7640 1 Processor -- start 2024-01-21T03:20:06.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.293+0000 7f9de1cd7640 1 -- start start 2024-01-21T03:20:06.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.293+0000 7f9de1cd7640 10 monclient: init 2024-01-21T03:20:06.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-01-21T03:20:06.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9ddc0a8a40 0x7f9ddc145b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 10 monclient(hunting): picked mon.noname-c con 0x7f9ddc0a8a40 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:06.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ddc0b2210 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9ddc0a8e20 0x7f9ddc14c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 10 monclient(hunting): picked mon.noname-a con 0x7f9ddc0a8e20 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc145700 con 0x7f9ddc0b2210 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ddc0aa7b0 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ddc0aa9b0 con 0x7f9ddc0a8a40 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9de1cd7640 10 monclient(hunting): authenticate will time out at 1352.609898s 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.294+0000 7f9ddbfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9ddc0a8e20 0x7f9ddc14c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.295+0000 7f9ddbfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9ddc0a8e20 0x7f9ddc14c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44404/0 (socket says 172.21.15.7:44404) 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.295+0000 7f9ddbfff640 1 -- 172.21.15.7:0/3349869509 learned_addr learned my addr 172.21.15.7:0/3349869509 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.295+0000 7f9ddbfff640 10 monclient(hunting): get_auth_request con 0x7f9ddc0a8e20 auth_method 0 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.295+0000 7f9ddbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.295+0000 7f9ddbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.295+0000 7f9ddbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.296+0000 7f9ddbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.296+0000 7f9ddbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.296+0000 7f9ddbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.298+0000 7f9ddbfff640 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2024-01-21T03:20:06.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.298+0000 7f9ddbfff640 1 -- 172.21.15.7:0/3349869509 >> v2:172.21.15.177:3300/0 conn(0x7f9ddc0a8a40 msgr2=0x7f9ddc145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:06.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.298+0000 7f9ddbfff640 1 --2- 172.21.15.7:0/3349869509 >> v2:172.21.15.177:3300/0 conn(0x7f9ddc0a8a40 0x7f9ddc145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.298+0000 7f9ddbfff640 1 -- 172.21.15.7:0/3349869509 >> v1:172.21.15.7:6789/0 conn(0x7f9ddc0b2210 legacy=0x7f9ddc13f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.298+0000 7f9ddbfff640 10 monclient: _finish_hunting 0 2024-01-21T03:20:06.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.298+0000 7f9ddbfff640 1 monclient: found mon.noname-a 2024-01-21T03:20:06.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.298+0000 7f9ddbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:06.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.298+0000 7f9ddbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:06.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.298+0000 7f9ddbfff640 1 -- 172.21.15.7:0/3349869509 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc0aac90 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.302+0000 7f9ddbfff640 1 --2- 172.21.15.7:0/3349869509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ddc0a8e20 0x7f9ddc14c8d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0b5320 tx=0x7f9dd402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e67145fc93922f00 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9dda7fc640 1 -- 172.21.15.7:0/3349869509 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd402cb20 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9dda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9dda7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9dda7fc640 10 monclient: dump: 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9dda7fc640 10 monclient: _finish_auth 0 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9dda7fc640 10 monclient: _check_auth_tickets 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9dda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9dda7fc640 1 -- 172.21.15.7:0/3349869509 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd402cc80 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9dda7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9dda7fc640 1 -- 172.21.15.7:0/3349869509 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40343f0 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9dda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9dda7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9dda7fc640 10 monclient: dump: 2024-01-21T03:20:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.295 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9de1cd7640 5 monclient: authenticate success, global_id 4314 2024-01-21T03:20:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9de1cd7640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9de1cd7640 4 set_mon_vals no callback set 2024-01-21T03:20:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9de1cd7640 10 monclient: shutdown 2024-01-21T03:20:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9de1cd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3349869509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ddc0a8e20 msgr2=0x7f9ddc14c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.306+0000 7f9de1cd7640 1 --2- 172.21.15.7:0/3349869509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ddc0a8e20 0x7f9ddc14c8d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0b5320 tx=0x7f9dd402bf80 comp rx=0 tx=0).stop 2024-01-21T03:20:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.307+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3349869509 shutdown_connections 2024-01-21T03:20:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.307+0000 7f9de1cd7640 1 --2- 172.21.15.7:0/3349869509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ddc0a8e20 0x7f9ddc14c8d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.307+0000 7f9de1cd7640 1 --2- 172.21.15.7:0/3349869509 >> v2:172.21.15.177:3300/0 conn(0x7f9ddc0a8a40 0x7f9ddc145b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.307+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3349869509 >> 172.21.15.7:0/3349869509 conn(0x7f9ddc066f10 msgr2=0x7f9ddc0b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.307+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3349869509 shutdown_connections 2024-01-21T03:20:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.307+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3349869509 wait complete. 2024-01-21T03:20:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.307+0000 7f9de1cd7640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.307+0000 7f9de1cd7640 10 monclient: monmap: 2024-01-21T03:20:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 1 librados: starting msgr at 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 1 librados: starting objecter 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 1 Processor -- start 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 1 -- start start 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 1 librados: setting wanted keys 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 1 librados: calling monclient init 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 10 monclient: init 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ddc0a8a40 0x7f9ddc144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 10 monclient(hunting): picked mon.noname-b con 0x7f9ddc0a8a40 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ddc0a8e20 0x7f9ddc140050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 10 monclient(hunting): picked mon.noname-a con 0x7f9ddc0a8e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ddc0b2210 0x7f9ddc133d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 10 monclient(hunting): picked mon.noname-c con 0x7f9ddc0b2210 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc135b10 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc144640 con 0x7f9ddc0a8a40 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc1447b0 con 0x7f9ddc0b2210 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.308+0000 7f9de1cd7640 10 monclient(hunting): authenticate will time out at 1352.624733s 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.309+0000 7f9ddaffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ddc0a8e20 0x7f9ddc140050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.309+0000 7f9ddbfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ddc0b2210 0x7f9ddc133d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.309+0000 7f9ddbfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ddc0b2210 0x7f9ddc133d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:46058/0 (socket says 172.21.15.7:46058) 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.309+0000 7f9ddbfff640 1 -- 172.21.15.7:0/3466940951 learned_addr learned my addr 172.21.15.7:0/3466940951 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.309+0000 7f9ddbfff640 10 monclient(hunting): get_auth_request con 0x7f9ddc0b2210 auth_method 0 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.309+0000 7f9ddbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.309+0000 7f9ddbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.309+0000 7f9ddbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.309+0000 7f9ddaffd640 10 monclient(hunting): get_auth_request con 0x7f9ddc0a8e20 auth_method 0 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.309+0000 7f9ddaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.309+0000 7f9ddaffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.309+0000 7f9ddaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.310+0000 7f9ddaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.310+0000 7f9ddaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.310+0000 7f9ddaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.310+0000 7f9ddbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.310+0000 7f9ddbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.310+0000 7f9ddbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.313+0000 7f9ddaffd640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-01-21T03:20:06.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.313+0000 7f9ddaffd640 1 -- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ddc0b2210 msgr2=0x7f9ddc133d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.313+0000 7f9ddaffd640 1 --2- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ddc0b2210 0x7f9ddc133d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.313+0000 7f9ddaffd640 1 -- 172.21.15.7:0/3466940951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ddc0a8a40 msgr2=0x7f9ddc144fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.313+0000 7f9ddaffd640 1 --2- 172.21.15.7:0/3466940951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ddc0a8a40 0x7f9ddc144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.313+0000 7f9ddaffd640 10 monclient: _finish_hunting 0 2024-01-21T03:20:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.313+0000 7f9ddaffd640 1 monclient: found mon.noname-a 2024-01-21T03:20:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.313+0000 7f9ddaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.313+0000 7f9ddaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.313+0000 7f9ddaffd640 1 -- 172.21.15.7:0/3466940951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd4009b30 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.313+0000 7f9ddbfff640 1 --2- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ddc0b2210 0x7f9ddc133d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T03:20:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.315+0000 7f9ddaffd640 1 --2- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ddc0a8e20 0x7f9ddc140050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c390 tx=0x7f9dd000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.315+0000 7f9dc3fff640 10 client.?.objecter ms_handle_connect 0x7f9ddc0a8e20 2024-01-21T03:20:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.315+0000 7f9dc3fff640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:06.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dc3fff640 1 -- 172.21.15.7:0/3466940951 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd00052d0 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dc3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:06.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dc3fff640 10 monclient: dump: 2024-01-21T03:20:06.305 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dc3fff640 10 monclient: _finish_auth 0 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dc3fff640 10 monclient: _check_auth_tickets 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dc3fff640 1 -- 172.21.15.7:0/3466940951 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd0005450 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dc3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dc3fff640 1 -- 172.21.15.7:0/3466940951 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0007260 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dc3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dc3fff640 10 monclient: dump: 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9de1cd7640 5 monclient: authenticate success, global_id 4320 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9de1cd7640 10 monclient: _renew_subs 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9de1cd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:06.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3466940951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ddc1355f0 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9dd9ffb640 4 set_mon_vals no callback set 2024-01-21T03:20:06.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9de1cd7640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:06.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9de1cd7640 10 monclient: _renew_subs 2024-01-21T03:20:06.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9de1cd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:06.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3466940951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ddc134470 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.319+0000 7f9de1cd7640 1 librados: init done 2024-01-21T03:20:06.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.320+0000 7f9de1cd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.320+0000 7f9de1cd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.320+0000 7f9de1cd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:06.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.320+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3466940951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ddc04e970 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-21T03:20:06.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-21T03:20:06.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.425+0000 7f330c505640 10 monclient: get_monmap_and_config 2024-01-21T03:20:06.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.425+0000 7f330c505640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.425+0000 7f330c505640 10 monclient: monmap: 2024-01-21T03:20:06.411 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.411 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.411 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:06.426244+0000 2024-01-21T03:20:06.411 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:06.426244+0000 2024-01-21T03:20:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:06.412 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.425+0000 7f330c505640 1 Processor -- start 2024-01-21T03:20:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.425+0000 7f330c505640 1 -- start start 2024-01-21T03:20:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.425+0000 7f330c505640 10 monclient: init 2024-01-21T03:20:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.426+0000 7f330c505640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.426+0000 7f330c505640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-01-21T03:20:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.426+0000 7f330c505640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33040a8a40 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.426+0000 7f330c505640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 0x7f330413f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.426+0000 7f330c505640 10 monclient(hunting): picked mon.noname-b con 0x7f33040b2210 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.426+0000 7f330c505640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33040a8e20 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.426+0000 7f330c505640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3304145700 con 0x7f33040a8a40 2024-01-21T03:20:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.426+0000 7f330c505640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33040aa9b0 con 0x7f33040a8e20 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.426+0000 7f330c505640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.426+0000 7f330c505640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33040aa7b0 con 0x7f33040b2210 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.426+0000 7f330c505640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.426+0000 7f330c505640 10 monclient(hunting): authenticate will time out at 1352.742428s 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.427+0000 7f3309aa0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 0x7f330413f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.427+0000 7f3309aa0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 0x7f330413f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:46064/0 (socket says 172.21.15.7:46064) 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.427+0000 7f3309aa0640 1 -- 172.21.15.7:0/2144761066 learned_addr learned my addr 172.21.15.7:0/2144761066 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.427+0000 7f3309aa0640 10 monclient(hunting): get_auth_request con 0x7f33040b2210 auth_method 0 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.427+0000 7f3309aa0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.427+0000 7f3309aa0640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.427+0000 7f3309aa0640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.428+0000 7f3309aa0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.428+0000 7f3309aa0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.428+0000 7f3309aa0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.431+0000 7f3309aa0640 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2024-01-21T03:20:06.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.431+0000 7f3309aa0640 1 -- 172.21.15.7:0/2144761066 >> v1:172.21.15.177:6789/0 conn(0x7f33040a8e20 legacy=0x7f330414c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:06.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.431+0000 7f3309aa0640 1 -- 172.21.15.7:0/2144761066 >> v1:172.21.15.7:6789/0 conn(0x7f33040a8a40 legacy=0x7f3304145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:06.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.431+0000 7f3309aa0640 10 monclient: _finish_hunting 0 2024-01-21T03:20:06.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.431+0000 7f3309aa0640 1 monclient: found mon.noname-b 2024-01-21T03:20:06.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.431+0000 7f3309aa0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:06.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.431+0000 7f3309aa0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:06.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.431+0000 7f3309aa0640 1 -- 172.21.15.7:0/2144761066 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33040aac90 con 0x7f33040b2210 2024-01-21T03:20:06.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.433+0000 7f3309aa0640 1 --2- 172.21.15.7:0/2144761066 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 0x7f330413f1c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f32f40098d0 tx=0x7f32f402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ebc756cb291ce39 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330929f640 1 -- 172.21.15.7:0/2144761066 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4031830 con 0x7f33040b2210 2024-01-21T03:20:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330929f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330929f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330929f640 10 monclient: dump: 2024-01-21T03:20:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.425 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330929f640 10 monclient: _finish_auth 0 2024-01-21T03:20:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330929f640 10 monclient: _check_auth_tickets 2024-01-21T03:20:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330929f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330929f640 1 -- 172.21.15.7:0/2144761066 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f4031990 con 0x7f33040b2210 2024-01-21T03:20:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330929f640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330c505640 5 monclient: authenticate success, global_id 4307 2024-01-21T03:20:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330929f640 1 -- 172.21.15.7:0/2144761066 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4031d20 con 0x7f33040b2210 2024-01-21T03:20:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330c505640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330c505640 4 set_mon_vals no callback set 2024-01-21T03:20:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330929f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330929f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.438+0000 7f330929f640 10 monclient: dump: 2024-01-21T03:20:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.427 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.439+0000 7f330c505640 10 monclient: shutdown 2024-01-21T03:20:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.439+0000 7f330c505640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.439+0000 7f330c505640 1 -- 172.21.15.7:0/2144761066 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 msgr2=0x7f330413f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.439+0000 7f330c505640 1 --2- 172.21.15.7:0/2144761066 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 0x7f330413f1c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f32f40098d0 tx=0x7f32f402b8d0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.439+0000 7f330c505640 1 -- 172.21.15.7:0/2144761066 shutdown_connections 2024-01-21T03:20:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.439+0000 7f330c505640 1 --2- 172.21.15.7:0/2144761066 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 0x7f330413f1c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.439+0000 7f330c505640 1 -- 172.21.15.7:0/2144761066 >> 172.21.15.7:0/2144761066 conn(0x7f3304066f10 msgr2=0x7f33040b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.439+0000 7f330c505640 1 -- 172.21.15.7:0/2144761066 shutdown_connections 2024-01-21T03:20:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.439+0000 7f330c505640 1 -- 172.21.15.7:0/2144761066 wait complete. 2024-01-21T03:20:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.440+0000 7f330c505640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.440+0000 7f330c505640 10 monclient: monmap: 2024-01-21T03:20:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.440+0000 7f330c505640 1 librados: starting msgr at 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.440+0000 7f330c505640 1 librados: starting objecter 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.441+0000 7f330c505640 1 Processor -- start 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.441+0000 7f330c505640 1 -- start start 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.441+0000 7f330c505640 1 librados: setting wanted keys 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.441+0000 7f330c505640 1 librados: calling monclient init 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.441+0000 7f330c505640 10 monclient: init 2024-01-21T03:20:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.441+0000 7f330c505640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.441+0000 7f330c505640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.441+0000 7f330c505640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33040a8a40 0x7f33041cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330c505640 10 monclient(hunting): picked mon.noname-b con 0x7f33040a8a40 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330c505640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33040a8e20 0x7f3304137a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330c505640 10 monclient(hunting): picked mon.noname-a con 0x7f33040a8e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330c505640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 0x7f3304137fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330c505640 10 monclient(hunting): picked mon.noname-c con 0x7f33040b2210 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330c505640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330c505640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33041384e0 con 0x7f33040a8e20 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330c505640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330c505640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33041cdd30 con 0x7f33040a8a40 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330c505640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330c505640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33041cc310 con 0x7f33040b2210 2024-01-21T03:20:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330c505640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330c505640 10 monclient(hunting): authenticate will time out at 1352.757808s 2024-01-21T03:20:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.442+0000 7f330aaa2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 0x7f3304137fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.443+0000 7f330aaa2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 0x7f3304137fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:46070/0 (socket says 172.21.15.7:46070) 2024-01-21T03:20:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.443+0000 7f330aaa2640 1 -- 172.21.15.7:0/2087811721 learned_addr learned my addr 172.21.15.7:0/2087811721 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.443+0000 7f330aaa2640 10 monclient(hunting): get_auth_request con 0x7f33040b2210 auth_method 0 2024-01-21T03:20:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.443+0000 7f330aaa2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.443+0000 7f330aaa2640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.443+0000 7f330aaa2640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.443+0000 7f330aaa2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.443+0000 7f330aaa2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.443+0000 7f330aaa2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.448+0000 7f330aaa2640 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2024-01-21T03:20:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.448+0000 7f330aaa2640 1 -- 172.21.15.7:0/2087811721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33040a8a40 msgr2=0x7f33041cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.448+0000 7f330aaa2640 1 --2- 172.21.15.7:0/2087811721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33040a8a40 0x7f33041cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.448+0000 7f330aaa2640 1 -- 172.21.15.7:0/2087811721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33040a8e20 msgr2=0x7f3304137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.448+0000 7f330aaa2640 1 --2- 172.21.15.7:0/2087811721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33040a8e20 0x7f3304137a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.448+0000 7f330aaa2640 10 monclient: _finish_hunting 0 2024-01-21T03:20:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.448+0000 7f330aaa2640 1 monclient: found mon.noname-c 2024-01-21T03:20:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.448+0000 7f330aaa2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:06.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.448+0000 7f330aaa2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:20:06.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.448+0000 7f330aaa2640 1 -- 172.21.15.7:0/2087811721 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f40094d0 con 0x7f33040b2210 2024-01-21T03:20:06.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.451+0000 7f330aaa2640 1 --2- 172.21.15.7:0/2087811721 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 0x7f3304137fa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f32fc00c7b0 tx=0x7f32fc00cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.451+0000 7f32faffd640 10 client.?.objecter ms_handle_connect 0x7f33040b2210 2024-01-21T03:20:06.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.451+0000 7f32faffd640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:06.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f32faffd640 1 -- 172.21.15.7:0/2087811721 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc003d80 con 0x7f33040b2210 2024-01-21T03:20:06.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f32faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f32faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:06.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f32faffd640 10 monclient: dump: 2024-01-21T03:20:06.443 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.443 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.443 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.443 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.443 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.443 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f32faffd640 10 monclient: _finish_auth 0 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f32faffd640 10 monclient: _check_auth_tickets 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f32faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f32faffd640 1 -- 172.21.15.7:0/2087811721 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32fc00ce60 con 0x7f33040b2210 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f32faffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f32faffd640 1 -- 172.21.15.7:0/2087811721 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc0052b0 con 0x7f33040b2210 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f330c505640 5 monclient: authenticate success, global_id 4310 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f330c505640 10 monclient: _renew_subs 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f330c505640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f3308a9e640 4 set_mon_vals no callback set 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f330c505640 1 -- 172.21.15.7:0/2087811721 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33041cd3b0 con 0x7f33040b2210 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f32faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f330c505640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f32faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f32faffd640 10 monclient: dump: 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f330c505640 10 monclient: _renew_subs 2024-01-21T03:20:06.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f330c505640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:06.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f330c505640 1 -- 172.21.15.7:0/2087811721 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33041cbb10 con 0x7f33040b2210 2024-01-21T03:20:06.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.456+0000 7f330c505640 1 librados: init done 2024-01-21T03:20:06.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.458+0000 7f330c505640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.458+0000 7f330c505640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.458+0000 7f330c505640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:06.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.458+0000 7f330c505640 1 -- 172.21.15.7:0/2087811721 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33041d76a0 con 0x7f33040b2210 2024-01-21T03:20:06.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.483+0000 7f32faffd640 1 -- 172.21.15.7:0/2087811721 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f32fc005410 con 0x7f33040b2210 2024-01-21T03:20:06.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.483+0000 7f820fa07640 10 monclient: get_monmap_and_config 2024-01-21T03:20:06.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.483+0000 7f820fa07640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.483+0000 7f820fa07640 10 monclient: monmap: 2024-01-21T03:20:06.469 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.469 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.470 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:06.484398+0000 2024-01-21T03:20:06.470 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:06.484398+0000 2024-01-21T03:20:06.470 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.470 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.470 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:06.470 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:06.470 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:06.470 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:06.470 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:06.470 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.483+0000 7f32faffd640 1 --2- 172.21.15.7:0/2087811721 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f32e806c8b0 0x7f32e806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 1 Processor -- start 2024-01-21T03:20:06.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f330a2a1640 1 --2- 172.21.15.7:0/2087811721 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f32e806c8b0 0x7f32e806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f330a2a1640 10 monclient: get_auth_request con 0x7f32e806c8b0 auth_method 0 2024-01-21T03:20:06.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 1 -- start start 2024-01-21T03:20:06.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 10 monclient: init 2024-01-21T03:20:06.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f32faffd640 10 client.4310.objecter ms_handle_connect 0x7f32e806c8b0 2024-01-21T03:20:06.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f330a2a1640 1 --2- 172.21.15.7:0/2087811721 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f32e806c8b0 0x7f32e806ed70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3300001560 tx=0x7f3300007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-01-21T03:20:06.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f82080a8b20 0x7f8208145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 10 monclient(hunting): picked mon.noname-c con 0x7f82080a8b20 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:06.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82080a9d10 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:06.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f82080b2310 0x7f820813f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 10 monclient(hunting): picked mon.noname-a con 0x7f82080b2310 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:06.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82081456f0 con 0x7f82080a9d10 2024-01-21T03:20:06.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82080a98c0 con 0x7f82080b2310 2024-01-21T03:20:06.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82081460c0 con 0x7f82080a8b20 2024-01-21T03:20:06.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.484+0000 7f820fa07640 10 monclient(hunting): authenticate will time out at 1352.800705s 2024-01-21T03:20:06.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.495+0000 7f982affd640 1 -- 172.21.15.7:0/1233089965 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9824054190 con 0x7f98340a8ab0 2024-01-21T03:20:06.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.495+0000 7f982affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.495+0000 7f982affd640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:06.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.499+0000 7f820d7a3640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f82080a8b20 0x7f8208145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.500+0000 7f820d7a3640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f82080a8b20 0x7f8208145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:56692/0 (socket says 172.21.15.7:56692) 2024-01-21T03:20:06.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.500+0000 7f820d7a3640 1 -- 172.21.15.7:0/2648267227 learned_addr learned my addr 172.21.15.7:0/2648267227 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:06.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.500+0000 7f820d7a3640 10 monclient(hunting): get_auth_request con 0x7f82080a8b20 auth_method 0 2024-01-21T03:20:06.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.500+0000 7f820d7a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.500+0000 7f820d7a3640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.500+0000 7f820d7a3640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.501+0000 7f820d7a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.501+0000 7f820d7a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.501+0000 7f820d7a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.503+0000 7f820d7a3640 10 monclient(hunting): handle_auth_done global_id 4357 payload 274 2024-01-21T03:20:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.503+0000 7f820d7a3640 1 -- 172.21.15.7:0/2648267227 >> v2:172.21.15.7:3300/0 conn(0x7f82080b2310 msgr2=0x7f820813f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.503+0000 7f820d7a3640 1 --2- 172.21.15.7:0/2648267227 >> v2:172.21.15.7:3300/0 conn(0x7f82080b2310 0x7f820813f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.503+0000 7f820d7a3640 1 -- 172.21.15.7:0/2648267227 >> v1:172.21.15.7:6789/0 conn(0x7f82080a9d10 legacy=0x7f820813f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.503+0000 7f820d7a3640 10 monclient: _finish_hunting 0 2024-01-21T03:20:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.503+0000 7f820d7a3640 1 monclient: found mon.noname-c 2024-01-21T03:20:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.503+0000 7f820d7a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.503+0000 7f820d7a3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.503+0000 7f820d7a3640 1 -- 172.21.15.7:0/2648267227 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8208146230 con 0x7f82080a8b20 2024-01-21T03:20:06.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.506+0000 7f820d7a3640 1 --2- 172.21.15.7:0/2648267227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82080a8b20 0x7f8208145b80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f82080a8600 tx=0x7f820402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb71e87718e57892 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f81fffff640 1 -- 172.21.15.7:0/2648267227 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204031870 con 0x7f82080a8b20 2024-01-21T03:20:06.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f81fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f81fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:06.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f81fffff640 10 monclient: dump: 2024-01-21T03:20:06.495 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f81fffff640 10 monclient: _finish_auth 0 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f81fffff640 10 monclient: _check_auth_tickets 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f81fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f81fffff640 1 -- 172.21.15.7:0/2648267227 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82040319d0 con 0x7f82080a8b20 2024-01-21T03:20:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f81fffff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f820fa07640 5 monclient: authenticate success, global_id 4357 2024-01-21T03:20:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f81fffff640 1 -- 172.21.15.7:0/2648267227 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204031d60 con 0x7f82080a8b20 2024-01-21T03:20:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f81fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f81fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f81fffff640 10 monclient: dump: 2024-01-21T03:20:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.497 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.498 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f820fa07640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.509+0000 7f820fa07640 4 set_mon_vals no callback set 2024-01-21T03:20:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.510+0000 7f820fa07640 10 monclient: shutdown 2024-01-21T03:20:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.510+0000 7f820fa07640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.510+0000 7f820fa07640 1 -- 172.21.15.7:0/2648267227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82080a8b20 msgr2=0x7f8208145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.510+0000 7f820fa07640 1 --2- 172.21.15.7:0/2648267227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82080a8b20 0x7f8208145b80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f82080a8600 tx=0x7f820402b910 comp rx=0 tx=0).stop 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.510+0000 7f820fa07640 1 -- 172.21.15.7:0/2648267227 shutdown_connections 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.510+0000 7f820fa07640 1 --2- 172.21.15.7:0/2648267227 >> v2:172.21.15.7:3300/0 conn(0x7f82080b2310 0x7f820813f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.510+0000 7f820fa07640 1 --2- 172.21.15.7:0/2648267227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82080a8b20 0x7f8208145b80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.510+0000 7f820fa07640 1 -- 172.21.15.7:0/2648267227 >> 172.21.15.7:0/2648267227 conn(0x7f8208066fb0 msgr2=0x7f82080b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.510+0000 7f820fa07640 1 -- 172.21.15.7:0/2648267227 shutdown_connections 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.510+0000 7f820fa07640 1 -- 172.21.15.7:0/2648267227 wait complete. 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.511+0000 7f820fa07640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.511+0000 7f820fa07640 10 monclient: monmap: 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:06.501 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.511+0000 7f820fa07640 1 librados: starting msgr at 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.511+0000 7f820fa07640 1 librados: starting objecter 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.511+0000 7f820fa07640 1 Processor -- start 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.511+0000 7f820fa07640 1 -- start start 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.511+0000 7f820fa07640 1 librados: setting wanted keys 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.511+0000 7f820fa07640 1 librados: calling monclient init 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.511+0000 7f820fa07640 10 monclient: init 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82080a8b20 0x7f820813be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 10 monclient(hunting): picked mon.noname-a con 0x7f82080a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f82080a9d10 0x7f82081394c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 10 monclient(hunting): picked mon.noname-c con 0x7f82080a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82080b2310 0x7f8208139a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 10 monclient(hunting): picked mon.noname-b con 0x7f82080b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8208141a70 con 0x7f82080a8b20 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8208139f40 con 0x7f82080b2310 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8208141cb0 con 0x7f82080a9d10 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.512+0000 7f820fa07640 10 monclient(hunting): authenticate will time out at 1352.828247s 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.513+0000 7f820dfa4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82080b2310 0x7f8208139a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.513+0000 7f820dfa4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82080b2310 0x7f8208139a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:56708/0 (socket says 172.21.15.7:56708) 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.513+0000 7f820dfa4640 1 -- 172.21.15.7:0/3827353314 learned_addr learned my addr 172.21.15.7:0/3827353314 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.513+0000 7f820dfa4640 10 monclient(hunting): get_auth_request con 0x7f82080b2310 auth_method 0 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.513+0000 7f820dfa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.513+0000 7f820dfa4640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.513+0000 7f820dfa4640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.514+0000 7f820dfa4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.514+0000 7f820dfa4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.514+0000 7f820dfa4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.516+0000 7f820dfa4640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2024-01-21T03:20:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.516+0000 7f820dfa4640 1 -- 172.21.15.7:0/3827353314 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f82080a9d10 msgr2=0x7f82081394c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.516+0000 7f820dfa4640 1 --2- 172.21.15.7:0/3827353314 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f82080a9d10 0x7f82081394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.516+0000 7f820dfa4640 1 -- 172.21.15.7:0/3827353314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82080a8b20 msgr2=0x7f820813be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.516+0000 7f820dfa4640 1 --2- 172.21.15.7:0/3827353314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82080a8b20 0x7f820813be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.516+0000 7f820dfa4640 10 monclient: _finish_hunting 0 2024-01-21T03:20:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.516+0000 7f820dfa4640 1 monclient: found mon.noname-b 2024-01-21T03:20:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.516+0000 7f820dfa4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.516+0000 7f820dfa4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.516+0000 7f820dfa4640 1 -- 172.21.15.7:0/3827353314 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82040094d0 con 0x7f82080b2310 2024-01-21T03:20:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.519+0000 7f820dfa4640 1 --2- 172.21.15.7:0/3827353314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82080b2310 0x7f8208139a00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f820000b970 tx=0x7f820000be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.519+0000 7f81fdffb640 10 client.?.objecter ms_handle_connect 0x7f82080b2310 2024-01-21T03:20:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.519+0000 7f81fdffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f81fdffb640 1 -- 172.21.15.7:0/3827353314 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820000cb40 con 0x7f82080b2310 2024-01-21T03:20:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f81fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f81fdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f81fdffb640 10 monclient: dump: 2024-01-21T03:20:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f81fdffb640 10 monclient: _finish_auth 0 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f81fdffb640 10 monclient: _check_auth_tickets 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f81fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f820fa07640 5 monclient: authenticate success, global_id 4360 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f820fa07640 10 monclient: _renew_subs 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f820fa07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f820fa07640 1 -- 172.21.15.7:0/3827353314 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8208140280 con 0x7f82080b2310 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f820fa07640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f820fa07640 10 monclient: _renew_subs 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f820fa07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f820fa07640 1 -- 172.21.15.7:0/3827353314 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8208140a20 con 0x7f82080b2310 2024-01-21T03:20:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.521+0000 7f820fa07640 1 librados: init done 2024-01-21T03:20:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.522+0000 7f820fa07640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.522+0000 7f820fa07640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.522+0000 7f820fa07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.522+0000 7f820fa07640 1 -- 172.21.15.7:0/3827353314 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82081404a0 con 0x7f82080b2310 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.522+0000 7f81fdffb640 1 -- 172.21.15.7:0/3827353314 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820000cca0 con 0x7f82080b2310 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.522+0000 7f81fdffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.522+0000 7f81fdffb640 1 -- 172.21.15.7:0/3827353314 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820001e660 con 0x7f82080b2310 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.522+0000 7f81fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.522+0000 7f81fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.522+0000 7f81fdffb640 10 monclient: dump: 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.522+0000 7f81ff7fe640 4 set_mon_vals no callback set 2024-01-21T03:20:06.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-21T03:20:06.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-21T03:20:06.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.589+0000 7f9dc3fff640 1 -- 172.21.15.7:0/3466940951 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9dd00073c0 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.590+0000 7f9dc3fff640 1 --2- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f9dc806c8e0 0x7f9dc806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.590+0000 7f9ddb7fe640 1 --2- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f9dc806c8e0 0x7f9dc806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.590+0000 7f9ddb7fe640 10 monclient: get_auth_request con 0x7f9dc806c8e0 auth_method 0 2024-01-21T03:20:06.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.590+0000 7f9dc3fff640 10 client.4320.objecter ms_handle_connect 0x7f9dc806c8e0 2024-01-21T03:20:06.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.590+0000 7f9ddb7fe640 1 --2- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f9dc806c8e0 0x7f9dc806eda0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc009710 tx=0x7f9dcc006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.609+0000 7f9dc3fff640 1 -- 172.21.15.7:0/3466940951 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9dd0057930 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.609+0000 7f9dc3fff640 10 client.4320.objecter ms_dispatch 0x7f9ddc000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:06.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.609+0000 7f9dc3fff640 3 client.4320.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:06.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.609+0000 7f9dc3fff640 3 client.4320.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:06.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.609+0000 7f9dc3fff640 20 client.4320.objecter dump_active .. 0 homeless 2024-01-21T03:20:06.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.609+0000 7f9dc3fff640 1 -- 172.21.15.7:0/3466940951 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9dd0053c70 con 0x7f9ddc0a8e20 2024-01-21T03:20:06.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.610+0000 7f9dc3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.610+0000 7f9dc3fff640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:06.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-21T03:20:06.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.678+0000 7fde515d0640 10 monclient: get_monmap_and_config 2024-01-21T03:20:06.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.678+0000 7fde515d0640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.678+0000 7fde515d0640 10 monclient: monmap: 2024-01-21T03:20:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:06.679817+0000 2024-01-21T03:20:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:06.679817+0000 2024-01-21T03:20:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:06.666 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.679+0000 7fde515d0640 1 Processor -- start 2024-01-21T03:20:06.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.679+0000 7fde515d0640 1 -- start start 2024-01-21T03:20:06.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.679+0000 7fde515d0640 10 monclient: init 2024-01-21T03:20:06.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.680+0000 7fde515d0640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.680+0000 7fde515d0640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-01-21T03:20:06.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.680+0000 7fde515d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde4c0a8b20 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:06.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.680+0000 7fde515d0640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fde4c0a9d10 0x7fde4c13f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.680+0000 7fde515d0640 10 monclient(hunting): picked mon.noname-a con 0x7fde4c0a9d10 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:06.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.680+0000 7fde515d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde4c0b2310 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:06.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.680+0000 7fde515d0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c1456f0 con 0x7fde4c0a8b20 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.680+0000 7fde515d0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c0a98c0 con 0x7fde4c0b2310 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.680+0000 7fde515d0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.680+0000 7fde515d0640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde4c146290 con 0x7fde4c0a9d10 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.680+0000 7fde515d0640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.680+0000 7fde515d0640 10 monclient(hunting): authenticate will time out at 1352.996388s 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.680+0000 7fde4a7fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fde4c0a9d10 0x7fde4c13f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.681+0000 7fde4affd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fde4c0a8b20 0x7fde4c145b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33724/0 (socket says 172.21.15.7:33724) 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.681+0000 7fde4affd640 1 -- 172.21.15.7:0/3370689277 learned_addr learned my addr 172.21.15.7:0/3370689277 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.681+0000 7fde4a7fc640 10 monclient(hunting): get_auth_request con 0x7fde4c0a9d10 auth_method 0 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.681+0000 7fde4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.681+0000 7fde4a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.681+0000 7fde4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.682+0000 7fde4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.682+0000 7fde4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.682+0000 7fde4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.684+0000 7fde4a7fc640 10 monclient(hunting): handle_auth_done global_id 4323 payload 274 2024-01-21T03:20:06.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.684+0000 7fde4a7fc640 1 -- 172.21.15.7:0/3370689277 >> v1:172.21.15.177:6789/0 conn(0x7fde4c0b2310 legacy=0x7fde4c13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:06.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.684+0000 7fde4a7fc640 1 -- 172.21.15.7:0/3370689277 >> v1:172.21.15.7:6789/0 conn(0x7fde4c0a8b20 legacy=0x7fde4c145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.684+0000 7fde4a7fc640 10 monclient: _finish_hunting 0 2024-01-21T03:20:06.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.684+0000 7fde4a7fc640 1 monclient: found mon.noname-a 2024-01-21T03:20:06.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.684+0000 7fde4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:06.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.684+0000 7fde4a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:06.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.684+0000 7fde4a7fc640 1 -- 172.21.15.7:0/3370689277 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c1463d0 con 0x7fde4c0a9d10 2024-01-21T03:20:06.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.690+0000 7f32faffd640 1 -- 172.21.15.7:0/2087811721 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f32fc058400 con 0x7f33040b2210 2024-01-21T03:20:06.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.690+0000 7f32faffd640 10 client.4310.objecter ms_dispatch 0x7f3304000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:06.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.690+0000 7f32faffd640 3 client.4310.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:06.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.690+0000 7f32faffd640 3 client.4310.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:06.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.690+0000 7f32faffd640 20 client.4310.objecter dump_active .. 0 homeless 2024-01-21T03:20:06.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.691+0000 7fde4a7fc640 1 --2- 172.21.15.7:0/3370689277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c0a9d10 0x7fde4c13f080 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fde400097a0 tx=0x7fde4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae05bf0bbe6edb84 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.696+0000 7f9838947640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T03:20:06.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.696+0000 7f9838947640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T03:20:06.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.696+0000 7f9838947640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.696+0000 7f9838947640 1 -- 172.21.15.7:0/1233089965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f9834072860 con 0x7f98340a8ab0 2024-01-21T03:20:06.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde49ffb640 1 -- 172.21.15.7:0/3370689277 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40031830 con 0x7fde4c0a9d10 2024-01-21T03:20:06.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde49ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:06.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde49ffb640 10 monclient: dump: 2024-01-21T03:20:06.685 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.685 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde49ffb640 10 monclient: _finish_auth 0 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde49ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde49ffb640 1 -- 172.21.15.7:0/3370689277 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde40031990 con 0x7fde4c0a9d10 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde49ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde49ffb640 1 -- 172.21.15.7:0/3370689277 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40031d20 con 0x7fde4c0a9d10 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde49ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:06.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde49ffb640 10 monclient: dump: 2024-01-21T03:20:06.687 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.687 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.687 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.687 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.687 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.687 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.687 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.687 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.687 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.687 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde515d0640 5 monclient: authenticate success, global_id 4323 2024-01-21T03:20:06.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde515d0640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:06.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.699+0000 7fde515d0640 4 set_mon_vals no callback set 2024-01-21T03:20:06.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.700+0000 7fde515d0640 10 monclient: shutdown 2024-01-21T03:20:06.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.700+0000 7fde515d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.700+0000 7fde515d0640 1 -- 172.21.15.7:0/3370689277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c0a9d10 msgr2=0x7fde4c13f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.700+0000 7fde515d0640 1 --2- 172.21.15.7:0/3370689277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c0a9d10 0x7fde4c13f080 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fde400097a0 tx=0x7fde4002b8d0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.700+0000 7fde515d0640 1 -- 172.21.15.7:0/3370689277 shutdown_connections 2024-01-21T03:20:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.700+0000 7fde515d0640 1 --2- 172.21.15.7:0/3370689277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c0a9d10 0x7fde4c13f080 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.700+0000 7fde515d0640 1 -- 172.21.15.7:0/3370689277 >> 172.21.15.7:0/3370689277 conn(0x7fde4c066fb0 msgr2=0x7fde4c0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.700+0000 7fde515d0640 1 -- 172.21.15.7:0/3370689277 shutdown_connections 2024-01-21T03:20:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.700+0000 7fde515d0640 1 -- 172.21.15.7:0/3370689277 wait complete. 2024-01-21T03:20:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.701+0000 7fde515d0640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.701+0000 7fde515d0640 10 monclient: monmap: 2024-01-21T03:20:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:06.692 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.701+0000 7fde515d0640 1 librados: starting msgr at 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.701+0000 7fde515d0640 1 librados: starting objecter 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.702+0000 7fde515d0640 1 Processor -- start 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.702+0000 7fde515d0640 1 -- start start 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.702+0000 7fde515d0640 1 librados: setting wanted keys 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.702+0000 7fde515d0640 1 librados: calling monclient init 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.702+0000 7fde515d0640 10 monclient: init 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.703+0000 7fde515d0640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.703+0000 7fde515d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T03:20:06.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.703+0000 7fde515d0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde4c0a8b20 0x7fde4c136fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.703+0000 7fde515d0640 10 monclient(hunting): picked mon.noname-b con 0x7fde4c0a8b20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.703+0000 7fde515d0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c0a9d10 0x7fde4c13fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.703+0000 7fde515d0640 10 monclient(hunting): picked mon.noname-a con 0x7fde4c0a9d10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.703+0000 7fde515d0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde4c0b2310 0x7fde4c1daa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.703+0000 7fde515d0640 10 monclient(hunting): picked mon.noname-c con 0x7fde4c0b2310 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.703+0000 7fde515d0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.703+0000 7fde515d0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c138170 con 0x7fde4c0a9d10 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.703+0000 7fde515d0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.704+0000 7fde515d0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c1376c0 con 0x7fde4c0a8b20 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.704+0000 7fde515d0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.704+0000 7fde515d0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c138350 con 0x7fde4c0b2310 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.704+0000 7fde515d0640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.704+0000 7fde515d0640 10 monclient(hunting): authenticate will time out at 1353.019860s 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.704+0000 7fde4a7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c0a9d10 0x7fde4c13fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.704+0000 7fde4a7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c0a9d10 0x7fde4c13fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45684/0 (socket says 172.21.15.7:45684) 2024-01-21T03:20:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.704+0000 7fde4a7fc640 1 -- 172.21.15.7:0/1836678464 learned_addr learned my addr 172.21.15.7:0/1836678464 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.704+0000 7fde4a7fc640 10 monclient(hunting): get_auth_request con 0x7fde4c0a9d10 auth_method 0 2024-01-21T03:20:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.704+0000 7fde4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.704+0000 7fde4a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.704+0000 7fde4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.705+0000 7fde4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.705+0000 7fde4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.705+0000 7fde4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.709+0000 7fde4a7fc640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2024-01-21T03:20:06.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.709+0000 7fde4a7fc640 1 -- 172.21.15.7:0/1836678464 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde4c0b2310 msgr2=0x7fde4c1daa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.709+0000 7fde4a7fc640 1 --2- 172.21.15.7:0/1836678464 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde4c0b2310 0x7fde4c1daa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.709+0000 7fde4a7fc640 1 -- 172.21.15.7:0/1836678464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde4c0a8b20 msgr2=0x7fde4c136fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.709+0000 7fde4a7fc640 1 --2- 172.21.15.7:0/1836678464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde4c0a8b20 0x7fde4c136fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.709+0000 7fde4a7fc640 10 monclient: _finish_hunting 0 2024-01-21T03:20:06.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.709+0000 7fde4a7fc640 1 monclient: found mon.noname-a 2024-01-21T03:20:06.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.709+0000 7fde4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:06.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.709+0000 7fde4a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:06.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.709+0000 7fde4a7fc640 1 -- 172.21.15.7:0/1836678464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde40009400 con 0x7fde4c0a9d10 2024-01-21T03:20:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.712+0000 7fde4a7fc640 1 --2- 172.21.15.7:0/1836678464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c0a9d10 0x7fde4c13fd60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fde400098c0 tx=0x7fde400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.712+0000 7fde337fe640 10 client.?.objecter ms_handle_connect 0x7fde4c0a9d10 2024-01-21T03:20:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.712+0000 7fde337fe640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7ff327661640 10 monclient: get_monmap_and_config 2024-01-21T03:20:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7ff327661640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7ff327661640 10 monclient: monmap: 2024-01-21T03:20:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:06.718428+0000 2024-01-21T03:20:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:06.718428+0000 2024-01-21T03:20:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:06.704 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:06.704 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:06.704 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde337fe640 1 -- 172.21.15.7:0/1836678464 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4003bb70 con 0x7fde4c0a9d10 2024-01-21T03:20:06.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde337fe640 10 monclient: dump: 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde337fe640 10 monclient: _finish_auth 0 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde337fe640 10 monclient: _check_auth_tickets 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde337fe640 1 -- 172.21.15.7:0/1836678464 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4003bcf0 con 0x7fde4c0a9d10 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde337fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde337fe640 1 -- 172.21.15.7:0/1836678464 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40045630 con 0x7fde4c0a9d10 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde337fe640 10 monclient: dump: 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde515d0640 5 monclient: authenticate success, global_id 4329 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde515d0640 10 monclient: _renew_subs 2024-01-21T03:20:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde515d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde515d0640 1 -- 172.21.15.7:0/1836678464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4c138630 con 0x7fde4c0a9d10 2024-01-21T03:20:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde497fa640 4 set_mon_vals no callback set 2024-01-21T03:20:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde515d0640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde515d0640 10 monclient: _renew_subs 2024-01-21T03:20:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde515d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde515d0640 1 -- 172.21.15.7:0/1836678464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4c142c10 con 0x7fde4c0a9d10 2024-01-21T03:20:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.717+0000 7fde515d0640 1 librados: init done 2024-01-21T03:20:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.718+0000 7ff327661640 1 Processor -- start 2024-01-21T03:20:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.718+0000 7ff327661640 1 -- start start 2024-01-21T03:20:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.718+0000 7ff327661640 10 monclient: init 2024-01-21T03:20:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7fde515d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7fde515d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7fde515d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7fde515d0640 1 -- 172.21.15.7:0/1836678464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4c142e60 con 0x7fde4c0a9d10 2024-01-21T03:20:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-01-21T03:20:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff32006fe10 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff32013c300 0x7ff32013ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 10 monclient(hunting): picked mon.noname-c con 0x7ff32013c300 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff32013f160 0x7ff320149530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 10 monclient(hunting): picked mon.noname-a con 0x7ff32013f160 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff320071d80 con 0x7ff32006fe10 2024-01-21T03:20:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff320072060 con 0x7ff32013f160 2024-01-21T03:20:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff320072340 con 0x7ff32013c300 2024-01-21T03:20:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.719+0000 7ff327661640 10 monclient(hunting): authenticate will time out at 1353.035164s 2024-01-21T03:20:06.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.742+0000 7fde337fe640 1 -- 172.21.15.7:0/1836678464 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fde4004e450 con 0x7fde4c0a9d10 2024-01-21T03:20:06.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.743+0000 7fde337fe640 1 --2- 172.21.15.7:0/1836678464 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fde3806c8e0 0x7fde3806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.743+0000 7fde4affd640 1 --2- 172.21.15.7:0/1836678464 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fde3806c8e0 0x7fde3806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.743+0000 7fde4affd640 10 monclient: get_auth_request con 0x7fde3806c8e0 auth_method 0 2024-01-21T03:20:06.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.744+0000 7fde4affd640 1 --2- 172.21.15.7:0/1836678464 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fde3806c8e0 0x7fde3806eda0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde3c000f60 tx=0x7fde3c007a80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.744+0000 7fde337fe640 10 client.4329.objecter ms_handle_connect 0x7fde3806c8e0 2024-01-21T03:20:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.747+0000 7fde337fe640 1 -- 172.21.15.7:0/1836678464 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fde4007abe0 con 0x7fde4c0a9d10 2024-01-21T03:20:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.747+0000 7fde337fe640 10 client.4329.objecter ms_dispatch 0x7fde4c000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.747+0000 7fde337fe640 3 client.4329.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.747+0000 7fde337fe640 3 client.4329.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.747+0000 7fde337fe640 20 client.4329.objecter dump_active .. 0 homeless 2024-01-21T03:20:06.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-21T03:20:06.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.764+0000 7f81fdffb640 1 -- 172.21.15.7:0/3827353314 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8200014440 con 0x7f82080b2310 2024-01-21T03:20:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.765+0000 7f81fdffb640 1 --2- 172.21.15.7:0/3827353314 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f81ec06c860 0x7f81ec06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.765+0000 7f820d7a3640 1 --2- 172.21.15.7:0/3827353314 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f81ec06c860 0x7f81ec06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.765+0000 7f820d7a3640 10 monclient: get_auth_request con 0x7f81ec06c860 auth_method 0 2024-01-21T03:20:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.766+0000 7f820d7a3640 1 --2- 172.21.15.7:0/3827353314 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f81ec06c860 0x7f81ec06ed20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8204036000 tx=0x7f8204030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.766+0000 7f81fdffb640 10 client.4360.objecter ms_handle_connect 0x7f81ec06c860 2024-01-21T03:20:06.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.767+0000 7f81fdffb640 1 -- 172.21.15.7:0/3827353314 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8200058950 con 0x7f82080b2310 2024-01-21T03:20:06.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.767+0000 7f81fdffb640 10 client.4360.objecter ms_dispatch 0x7f8208000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:06.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.767+0000 7f81fdffb640 3 client.4360.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:06.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.767+0000 7f81fdffb640 3 client.4360.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:06.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.767+0000 7f81fdffb640 20 client.4360.objecter dump_active .. 0 homeless 2024-01-21T03:20:06.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.777+0000 7f81fdffb640 1 -- 172.21.15.7:0/3827353314 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8200054c90 con 0x7f82080b2310 2024-01-21T03:20:06.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.777+0000 7f81fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.777+0000 7f81fdffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:06.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.777+0000 7f4e7ebb0640 10 monclient: get_monmap_and_config 2024-01-21T03:20:06.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.777+0000 7f4e7ebb0640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.777+0000 7f4e7ebb0640 10 monclient: monmap: 2024-01-21T03:20:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:06.778676+0000 2024-01-21T03:20:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:06.778676+0000 2024-01-21T03:20:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:06.764 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.778+0000 7f4e7ebb0640 1 Processor -- start 2024-01-21T03:20:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.778+0000 7f4e7ebb0640 1 -- start start 2024-01-21T03:20:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.778+0000 7f4e7ebb0640 10 monclient: init 2024-01-21T03:20:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.779+0000 7f4e7ebb0640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.779+0000 7f4e7ebb0640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-01-21T03:20:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.779+0000 7f4e7ebb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e780b1bb0 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.779+0000 7f4e7ebb0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e7813f5c0 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.779+0000 7f4e7ebb0640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4e78145af0 0x7f4e7814d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.779+0000 7f4e7ebb0640 10 monclient(hunting): picked mon.noname-a con 0x7f4e78145af0 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.779+0000 7f4e7ebb0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e780acc70 con 0x7f4e780b1bb0 2024-01-21T03:20:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.780+0000 7f4e7ebb0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e780ae610 con 0x7f4e7813f5c0 2024-01-21T03:20:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.780+0000 7f4e7ebb0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.780+0000 7f4e7ebb0640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e780b1f90 con 0x7f4e78145af0 2024-01-21T03:20:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.780+0000 7f4e7ebb0640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.780+0000 7f4e7ebb0640 10 monclient(hunting): authenticate will time out at 1353.095781s 2024-01-21T03:20:06.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.785+0000 7ff3253fd640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7ff32006fe10 0x7ff32013bbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:36884/0 (socket says 172.21.15.7:36884) 2024-01-21T03:20:06.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.786+0000 7ff3253fd640 1 -- 172.21.15.7:0/2610386970 learned_addr learned my addr 172.21.15.7:0/2610386970 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:06.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.787+0000 7ff324bfc640 1 --2- 172.21.15.7:0/2610386970 >> v2:172.21.15.177:3300/0 conn(0x7ff32013c300 0x7ff32013ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.788+0000 7ff324bfc640 10 monclient(hunting): get_auth_request con 0x7ff32013c300 auth_method 0 2024-01-21T03:20:06.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.788+0000 7ff324bfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.788+0000 7ff324bfc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.788+0000 7ff324bfc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.789+0000 7ff324bfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.789+0000 7ff324bfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.790+0000 7ff324bfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.790+0000 7f4e77fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f4e7813f5c0 0x7f4e781453e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:36894/0 (socket says 172.21.15.7:36894) 2024-01-21T03:20:06.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.790+0000 7f4e77fff640 1 -- 172.21.15.7:0/2552683999 learned_addr learned my addr 172.21.15.7:0/2552683999 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:06.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.791+0000 7ff324bfc640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2024-01-21T03:20:06.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.791+0000 7ff324bfc640 1 -- 172.21.15.7:0/2610386970 >> v2:172.21.15.7:3300/0 conn(0x7ff32013f160 msgr2=0x7ff320149530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:06.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.791+0000 7ff324bfc640 1 --2- 172.21.15.7:0/2610386970 >> v2:172.21.15.7:3300/0 conn(0x7ff32013f160 0x7ff320149530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.791+0000 7ff324bfc640 1 -- 172.21.15.7:0/2610386970 >> v1:172.21.15.177:6789/0 conn(0x7ff32006fe10 legacy=0x7ff32013bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.791+0000 7ff324bfc640 10 monclient: _finish_hunting 0 2024-01-21T03:20:06.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.791+0000 7ff324bfc640 1 monclient: found mon.noname-c 2024-01-21T03:20:06.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.791+0000 7ff324bfc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:06.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.791+0000 7ff324bfc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.791+0000 7ff324bfc640 1 -- 172.21.15.7:0/2610386970 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff320072760 con 0x7ff32013c300 2024-01-21T03:20:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.797+0000 7f982affd640 1 -- 172.21.15.7:0/1233089965 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9824020090 con 0x7f98340a8ab0 2024-01-21T03:20:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.798+0000 7f982affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T03:20:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.798+0000 7f982affd640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:06.784 INFO:teuthology.orchestra.run.smithi007.stdout:77309411336 2024-01-21T03:20:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.799+0000 7ff324bfc640 1 --2- 172.21.15.7:0/2610386970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff32013c300 0x7ff32013ec20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff320071670 tx=0x7ff31002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe93fd1d51511a4c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.799+0000 7f9838947640 10 librados: watch_flush enter 2024-01-21T03:20:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 10 librados: watch_flush exit 2024-01-21T03:20:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 20 client.4354.objecter shutdown clearing up homeless session... 2024-01-21T03:20:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 10 client.4354.objecter successfully canceled tick 2024-01-21T03:20:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 1 -- 172.21.15.7:0/1233089965 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f981406c8b0 msgr2=0x7f981406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 1 --2- 172.21.15.7:0/1233089965 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f981406c8b0 0x7f981406ed70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98341db4b0 tx=0x7f981c009040 comp rx=0 tx=0).stop 2024-01-21T03:20:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 10 monclient: shutdown 2024-01-21T03:20:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 1 -- 172.21.15.7:0/1233089965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98340a8ab0 msgr2=0x7f98340ab9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 1 --2- 172.21.15.7:0/1233089965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98340a8ab0 0x7f98340ab9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f982400c400 tx=0x7f982400c8c0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 1 -- 172.21.15.7:0/1233089965 shutdown_connections 2024-01-21T03:20:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 1 --2- 172.21.15.7:0/1233089965 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f981406c8b0 0x7f981406ed70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 1 --2- 172.21.15.7:0/1233089965 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f983413f950 0x7f98340abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 1 --2- 172.21.15.7:0/1233089965 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98340a8e90 0x7f98341460c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 1 --2- 172.21.15.7:0/1233089965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98340a8ab0 0x7f98340ab9a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 1 -- 172.21.15.7:0/1233089965 >> 172.21.15.7:0/1233089965 conn(0x7f9834066f00 msgr2=0x7f983413d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 1 -- 172.21.15.7:0/1233089965 shutdown_connections 2024-01-21T03:20:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 1 -- 172.21.15.7:0/1233089965 wait complete. 2024-01-21T03:20:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.800+0000 7f9838947640 1 librados: shutdown 2024-01-21T03:20:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff317fff640 1 -- 172.21.15.7:0/2610386970 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff310031830 con 0x7ff32013c300 2024-01-21T03:20:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff317fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff317fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff317fff640 10 monclient: dump: 2024-01-21T03:20:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff317fff640 10 monclient: _finish_auth 0 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff317fff640 10 monclient: _check_auth_tickets 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff317fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff317fff640 1 -- 172.21.15.7:0/2610386970 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff310031990 con 0x7ff32013c300 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff327661640 5 monclient: authenticate success, global_id 4363 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff327661640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff317fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff317fff640 1 -- 172.21.15.7:0/2610386970 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff310031d20 con 0x7ff32013c300 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff317fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff317fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff317fff640 10 monclient: dump: 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff327661640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:06.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.804+0000 7ff327661640 4 set_mon_vals no callback set 2024-01-21T03:20:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.805+0000 7ff327661640 10 monclient: shutdown 2024-01-21T03:20:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.805+0000 7ff327661640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.805+0000 7ff327661640 1 -- 172.21.15.7:0/2610386970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff32013c300 msgr2=0x7ff32013ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.805+0000 7ff327661640 1 --2- 172.21.15.7:0/2610386970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff32013c300 0x7ff32013ec20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff320071670 tx=0x7ff31002b8d0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.805+0000 7ff327661640 1 -- 172.21.15.7:0/2610386970 shutdown_connections 2024-01-21T03:20:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.805+0000 7ff327661640 1 --2- 172.21.15.7:0/2610386970 >> v2:172.21.15.7:3300/0 conn(0x7ff32013f160 0x7ff320149530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.805+0000 7ff327661640 1 --2- 172.21.15.7:0/2610386970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff32013c300 0x7ff32013ec20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.805+0000 7ff327661640 1 -- 172.21.15.7:0/2610386970 >> 172.21.15.7:0/2610386970 conn(0x7ff3200b4910 msgr2=0x7ff3200b4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.805+0000 7ff327661640 1 -- 172.21.15.7:0/2610386970 shutdown_connections 2024-01-21T03:20:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.806+0000 7ff327661640 1 -- 172.21.15.7:0/2610386970 wait complete. 2024-01-21T03:20:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.807+0000 7ff327661640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.807+0000 7ff327661640 10 monclient: monmap: 2024-01-21T03:20:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.807+0000 7ff327661640 1 librados: starting msgr at 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.807+0000 7ff327661640 1 librados: starting objecter 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.807+0000 7ff327661640 1 Processor -- start 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.808+0000 7ff327661640 1 -- start start 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.808+0000 7ff327661640 1 librados: setting wanted keys 2024-01-21T03:20:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.808+0000 7ff327661640 1 librados: calling monclient init 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.808+0000 7ff327661640 10 monclient: init 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff32006fe10 0x7ff32013df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 10 monclient(hunting): picked mon.noname-b con 0x7ff32006fe10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff32013c300 0x7ff3201d68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 10 monclient(hunting): picked mon.noname-c con 0x7ff32013c300 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff32013f160 0x7ff3200adb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 10 monclient(hunting): picked mon.noname-a con 0x7ff32013f160 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3200a9270 con 0x7ff32013f160 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3201d6de0 con 0x7ff32006fe10 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3201d6fa0 con 0x7ff32013c300 2024-01-21T03:20:06.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff327661640 10 monclient(hunting): authenticate will time out at 1353.125576s 2024-01-21T03:20:06.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.809+0000 7ff3253fd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff32006fe10 0x7ff32013df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.810+0000 7ff3253fd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff32006fe10 0x7ff32013df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:56742/0 (socket says 172.21.15.7:56742) 2024-01-21T03:20:06.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.810+0000 7ff3253fd640 1 -- 172.21.15.7:0/20091628 learned_addr learned my addr 172.21.15.7:0/20091628 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:06.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.810+0000 7ff3253fd640 10 monclient(hunting): get_auth_request con 0x7ff32006fe10 auth_method 0 2024-01-21T03:20:06.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.810+0000 7ff3253fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:06.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.810+0000 7ff3253fd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:06.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.810+0000 7ff3253fd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:06.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.810+0000 7ff3253fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:06.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.810+0000 7ff3253fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:06.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.810+0000 7ff3253fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:06.801 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411337 got 77309411336 for osd.6 2024-01-21T03:20:06.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.813+0000 7ff3253fd640 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2024-01-21T03:20:06.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.813+0000 7ff3253fd640 1 -- 172.21.15.7:0/20091628 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff32013c300 msgr2=0x7ff3201d68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.813+0000 7ff3253fd640 1 --2- 172.21.15.7:0/20091628 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff32013c300 0x7ff3201d68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.813+0000 7ff3253fd640 1 -- 172.21.15.7:0/20091628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff32013f160 msgr2=0x7ff3200adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:06.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.813+0000 7ff3253fd640 1 --2- 172.21.15.7:0/20091628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff32013f160 0x7ff3200adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:06.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.813+0000 7ff3253fd640 10 monclient: _finish_hunting 0 2024-01-21T03:20:06.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.813+0000 7ff3253fd640 1 monclient: found mon.noname-b 2024-01-21T03:20:06.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.813+0000 7ff3253fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:06.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.813+0000 7ff3253fd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.813+0000 7ff3253fd640 1 -- 172.21.15.7:0/20091628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff310009400 con 0x7ff32006fe10 2024-01-21T03:20:06.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.815+0000 7ff3253fd640 1 --2- 172.21.15.7:0/20091628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff32006fe10 0x7ff32013df20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff31c00c4a0 tx=0x7ff31c00c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.815+0000 7ff315ffb640 10 client.?.objecter ms_handle_connect 0x7ff32006fe10 2024-01-21T03:20:06.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.815+0000 7ff315ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:06.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff315ffb640 1 -- 172.21.15.7:0/20091628 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c005ce0 con 0x7ff32006fe10 2024-01-21T03:20:06.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff315ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff315ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:06.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff315ffb640 10 monclient: dump: 2024-01-21T03:20:06.806 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.806 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.806 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.806 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff315ffb640 10 monclient: _finish_auth 0 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff315ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff315ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff315ffb640 1 -- 172.21.15.7:0/20091628 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c005e40 con 0x7ff32006fe10 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff315ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff315ffb640 1 -- 172.21.15.7:0/20091628 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c007270 con 0x7ff32006fe10 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff315ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff315ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff315ffb640 10 monclient: dump: 2024-01-21T03:20:06.807 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.818+0000 7ff3177fe640 4 set_mon_vals no callback set 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.819+0000 7ff327661640 5 monclient: authenticate success, global_id 4366 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.819+0000 7ff327661640 10 monclient: _renew_subs 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.819+0000 7ff327661640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.819+0000 7ff327661640 1 -- 172.21.15.7:0/20091628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3200ae040 con 0x7ff32006fe10 2024-01-21T03:20:06.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.819+0000 7ff327661640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:06.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.820+0000 7ff327661640 10 monclient: _renew_subs 2024-01-21T03:20:06.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.820+0000 7ff327661640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.820+0000 7ff327661640 1 -- 172.21.15.7:0/20091628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3200ae470 con 0x7ff32006fe10 2024-01-21T03:20:06.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.820+0000 7ff327661640 1 librados: init done 2024-01-21T03:20:06.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.821+0000 7ff327661640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.821+0000 7ff327661640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.821+0000 7ff327661640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:06.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.821+0000 7ff327661640 1 -- 172.21.15.7:0/20091628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3200ae6c0 con 0x7ff32006fe10 2024-01-21T03:20:06.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.839+0000 7ff315ffb640 1 -- 172.21.15.7:0/20091628 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ff31c0073d0 con 0x7ff32006fe10 2024-01-21T03:20:06.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.839+0000 7ff315ffb640 1 --2- 172.21.15.7:0/20091628 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ff304070cf0 0x7ff3040731b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.839+0000 7ff324bfc640 1 --2- 172.21.15.7:0/20091628 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ff304070cf0 0x7ff3040731b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:06.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.839+0000 7ff324bfc640 10 monclient: get_auth_request con 0x7ff304070cf0 auth_method 0 2024-01-21T03:20:06.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.840+0000 7ff324bfc640 1 --2- 172.21.15.7:0/20091628 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ff304070cf0 0x7ff3040731b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff310036000 tx=0x7ff310030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:06.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.840+0000 7ff315ffb640 10 client.4366.objecter ms_handle_connect 0x7ff304070cf0 2024-01-21T03:20:06.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.859+0000 7f32faffd640 1 -- 172.21.15.7:0/2087811721 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f32fc054040 con 0x7f33040b2210 2024-01-21T03:20:06.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.859+0000 7f32faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:06.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.859+0000 7f32faffd640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:06.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.905+0000 7f83c76a9640 10 monclient: get_monmap_and_config 2024-01-21T03:20:06.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.905+0000 7f83c76a9640 10 monclient: build_initial_monmap 2024-01-21T03:20:06.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.906+0000 7f83c76a9640 10 monclient: monmap: 2024-01-21T03:20:06.892 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:06.892 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:06.892 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:06.906990+0000 2024-01-21T03:20:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:06.906990+0000 2024-01-21T03:20:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:06.893 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.906+0000 7f83c76a9640 1 Processor -- start 2024-01-21T03:20:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.906+0000 7f83c76a9640 1 -- start start 2024-01-21T03:20:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.906+0000 7f83c76a9640 10 monclient: init 2024-01-21T03:20:06.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:06.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-01-21T03:20:06.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f83c00a8a40 0x7f83c0145b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 10 monclient(hunting): picked mon.noname-c con 0x7f83c00a8a40 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:06.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f83c00b2210 0x7f83c013f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:06.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 10 monclient(hunting): picked mon.noname-a con 0x7f83c00b2210 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:06.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83c00a8e20 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:06.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c0145700 con 0x7f83c00a8e20 2024-01-21T03:20:06.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c00aa7b0 con 0x7f83c00b2210 2024-01-21T03:20:06.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:06.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c00aa9b0 con 0x7f83c00a8a40 2024-01-21T03:20:06.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:06.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.907+0000 7f83c76a9640 10 monclient(hunting): authenticate will time out at 1353.223579s 2024-01-21T03:20:06.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.915+0000 7f9de1cd7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T03:20:06.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.915+0000 7f9de1cd7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T03:20:06.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.915+0000 7f9de1cd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:06.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:06.915+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3466940951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9ddc137460 con 0x7f9ddc0a8e20 2024-01-21T03:20:07.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.020+0000 7f820fa07640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T03:20:07.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.020+0000 7f820fa07640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T03:20:07.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.020+0000 7f820fa07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:07.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.020+0000 7f820fa07640 1 -- 172.21.15.7:0/3827353314 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f820804e970 con 0x7f82080b2310 2024-01-21T03:20:07.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.053+0000 7f4e7d14d640 1 --2- 172.21.15.7:0/2552683999 >> v2:172.21.15.7:3300/0 conn(0x7f4e78145af0 0x7f4e7814d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:07.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.055+0000 7f4e7d14d640 10 monclient(hunting): get_auth_request con 0x7f4e78145af0 auth_method 0 2024-01-21T03:20:07.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.055+0000 7f4e7d14d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:07.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.055+0000 7f4e7d14d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:07.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.055+0000 7f4e7d14d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:07.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.058+0000 7f83c4c44640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f83c00b2210 0x7f83c013f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:07.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.058+0000 7f4e7d14d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:07.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.058+0000 7f4e7d14d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:07.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.058+0000 7f4e7d14d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:07.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.060+0000 7f4e7d14d640 10 monclient(hunting): handle_auth_done global_id 4332 payload 274 2024-01-21T03:20:07.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.060+0000 7f4e7d14d640 1 -- 172.21.15.7:0/2552683999 >> v1:172.21.15.177:6789/0 conn(0x7f4e7813f5c0 legacy=0x7f4e781453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.060+0000 7f4e7d14d640 1 -- 172.21.15.7:0/2552683999 >> v1:172.21.15.7:6789/0 conn(0x7f4e780b1bb0 legacy=0x7f4e7813eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:07.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.060+0000 7f4e7d14d640 10 monclient: _finish_hunting 0 2024-01-21T03:20:07.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.060+0000 7f4e7d14d640 1 monclient: found mon.noname-a 2024-01-21T03:20:07.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.060+0000 7f4e7d14d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:07.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.060+0000 7f4e7d14d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:07.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.060+0000 7f4e7d14d640 1 -- 172.21.15.7:0/2552683999 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e780ae8f0 con 0x7f4e78145af0 2024-01-21T03:20:07.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.061+0000 7f83c4c44640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f83c00b2210 0x7f83c013f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45724/0 (socket says 172.21.15.7:45724) 2024-01-21T03:20:07.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.061+0000 7f83c4c44640 1 -- 172.21.15.7:0/3174506474 learned_addr learned my addr 172.21.15.7:0/3174506474 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:07.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.061+0000 7f83c4c44640 10 monclient(hunting): get_auth_request con 0x7f83c00b2210 auth_method 0 2024-01-21T03:20:07.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.061+0000 7f83c4c44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:07.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.061+0000 7f83c4c44640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:07.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.061+0000 7f83c4c44640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:07.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.062+0000 7f83c4c44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:07.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.062+0000 7f83c4c44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:07.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.062+0000 7f83c4c44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:07.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.063+0000 7ff315ffb640 1 -- 172.21.15.7:0/20091628 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff31c057be0 con 0x7ff32006fe10 2024-01-21T03:20:07.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.063+0000 7ff315ffb640 10 client.4366.objecter ms_dispatch 0x7ff320000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:07.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.063+0000 7ff315ffb640 3 client.4366.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:07.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.063+0000 7ff315ffb640 3 client.4366.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:07.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.063+0000 7ff315ffb640 20 client.4366.objecter dump_active .. 0 homeless 2024-01-21T03:20:07.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.073+0000 7ff315ffb640 1 -- 172.21.15.7:0/20091628 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ff31c053f20 con 0x7ff32006fe10 2024-01-21T03:20:07.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.073+0000 7ff315ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:07.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.073+0000 7ff315ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:07.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.080+0000 7fde337fe640 1 -- 172.21.15.7:0/1836678464 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fde40076820 con 0x7fde4c0a9d10 2024-01-21T03:20:07.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.080+0000 7fde337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:07.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.080+0000 7fde337fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:07.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.081+0000 7f81fdffb640 1 -- 172.21.15.7:0/3827353314 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8200023090 con 0x7f82080b2310 2024-01-21T03:20:07.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.081+0000 7f81fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T03:20:07.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.081+0000 7f81fdffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:07.068 INFO:teuthology.orchestra.run.smithi007.stdout:94489280516 2024-01-21T03:20:07.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.082+0000 7f83c5445640 1 --2- 172.21.15.7:0/3174506474 >> v2:172.21.15.177:3300/0 conn(0x7f83c00a8a40 0x7f83c0145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:07.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.082+0000 7f4e7d14d640 1 --2- 172.21.15.7:0/2552683999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e78145af0 0x7f4e7814d780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009d90 tx=0x7f4e7002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7eac2d4eb9cf01 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:07.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.083+0000 7f83c5445640 10 monclient(hunting): get_auth_request con 0x7f83c00a8a40 auth_method 0 2024-01-21T03:20:07.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.083+0000 7f83c5445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:07.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.083+0000 7f83c5445640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:07.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.083+0000 7f83c5445640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:07.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 10 librados: watch_flush enter 2024-01-21T03:20:07.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f83c5445640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:07.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f83c5445640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:07.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f83c5445640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:07.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 10 librados: watch_flush exit 2024-01-21T03:20:07.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 20 client.4360.objecter shutdown clearing up homeless session... 2024-01-21T03:20:07.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 10 client.4360.objecter successfully canceled tick 2024-01-21T03:20:07.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 1 -- 172.21.15.7:0/3827353314 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f81ec06c860 msgr2=0x7f81ec06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 1 --2- 172.21.15.7:0/3827353314 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f81ec06c860 0x7f81ec06ed20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8204036000 tx=0x7f8204030000 comp rx=0 tx=0).stop 2024-01-21T03:20:07.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 10 monclient: shutdown 2024-01-21T03:20:07.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:07.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 1 -- 172.21.15.7:0/3827353314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82080b2310 msgr2=0x7f8208139a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 1 --2- 172.21.15.7:0/3827353314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82080b2310 0x7f8208139a00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f820000b970 tx=0x7f820000be30 comp rx=0 tx=0).stop 2024-01-21T03:20:07.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 1 -- 172.21.15.7:0/3827353314 shutdown_connections 2024-01-21T03:20:07.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 1 --2- 172.21.15.7:0/3827353314 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f81ec06c860 0x7f81ec06ed20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 1 --2- 172.21.15.7:0/3827353314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82080b2310 0x7f8208139a00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 1 --2- 172.21.15.7:0/3827353314 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f82080a9d10 0x7f82081394c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.084+0000 7f820fa07640 1 --2- 172.21.15.7:0/3827353314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82080a8b20 0x7f820813be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.085+0000 7f820fa07640 1 -- 172.21.15.7:0/3827353314 >> 172.21.15.7:0/3827353314 conn(0x7f8208066fb0 msgr2=0x7f82080b6490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:07.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.085+0000 7f820fa07640 1 -- 172.21.15.7:0/3827353314 shutdown_connections 2024-01-21T03:20:07.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.085+0000 7f820fa07640 1 -- 172.21.15.7:0/3827353314 wait complete. 2024-01-21T03:20:07.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.085+0000 7f820fa07640 1 librados: shutdown 2024-01-21T03:20:07.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.085+0000 7f83c5445640 10 monclient(hunting): handle_auth_done global_id 4369 payload 274 2024-01-21T03:20:07.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.085+0000 7f83c5445640 1 -- 172.21.15.7:0/3174506474 >> v2:172.21.15.7:3300/0 conn(0x7f83c00b2210 msgr2=0x7f83c013f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:07.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.085+0000 7f83c5445640 1 --2- 172.21.15.7:0/3174506474 >> v2:172.21.15.7:3300/0 conn(0x7f83c00b2210 0x7f83c013f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.085+0000 7f83c5445640 1 -- 172.21.15.7:0/3174506474 >> v1:172.21.15.7:6789/0 conn(0x7f83c00a8e20 legacy=0x7f83c0141fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:07.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.086+0000 7f83c5445640 10 monclient: _finish_hunting 0 2024-01-21T03:20:07.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.086+0000 7f83c5445640 1 monclient: found mon.noname-c 2024-01-21T03:20:07.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.086+0000 7f83c5445640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:07.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.086+0000 7f83c5445640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:20:07.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.086+0000 7f83c5445640 1 -- 172.21.15.7:0/3174506474 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c00aac90 con 0x7f83c00a8a40 2024-01-21T03:20:07.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.088+0000 7f83c5445640 1 --2- 172.21.15.7:0/3174506474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83c00a8a40 0x7f83c0145b90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f83c00b5320 tx=0x7f83bc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7338149100bde358 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83b7fff640 1 -- 172.21.15.7:0/3174506474 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83bc02e070 con 0x7f83c00a8a40 2024-01-21T03:20:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83b7fff640 10 monclient: dump: 2024-01-21T03:20:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83b7fff640 10 monclient: _finish_auth 0 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83b7fff640 10 monclient: _check_auth_tickets 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83b7fff640 1 -- 172.21.15.7:0/3174506474 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83bc0315a0 con 0x7f83c00a8a40 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83c76a9640 5 monclient: authenticate success, global_id 4369 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83c76a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83b7fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83b7fff640 1 -- 172.21.15.7:0/3174506474 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83bc031930 con 0x7f83c00a8a40 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83c76a9640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83c76a9640 4 set_mon_vals no callback set 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.093+0000 7f83b7fff640 10 monclient: dump: 2024-01-21T03:20:07.081 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.094+0000 7f83c76a9640 10 monclient: shutdown 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.094+0000 7f83c76a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.094+0000 7f83c76a9640 1 -- 172.21.15.7:0/3174506474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83c00a8a40 msgr2=0x7f83c0145b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.094+0000 7f83c76a9640 1 --2- 172.21.15.7:0/3174506474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83c00a8a40 0x7f83c0145b90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f83c00b5320 tx=0x7f83bc02b840 comp rx=0 tx=0).stop 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.094+0000 7f83c76a9640 1 -- 172.21.15.7:0/3174506474 shutdown_connections 2024-01-21T03:20:07.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.094+0000 7f83c76a9640 1 --2- 172.21.15.7:0/3174506474 >> v2:172.21.15.7:3300/0 conn(0x7f83c00b2210 0x7f83c013f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.094+0000 7f83c76a9640 1 --2- 172.21.15.7:0/3174506474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83c00a8a40 0x7f83c0145b90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.094+0000 7f83c76a9640 1 -- 172.21.15.7:0/3174506474 >> 172.21.15.7:0/3174506474 conn(0x7f83c0066f10 msgr2=0x7f83c00b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:07.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.094+0000 7f83c76a9640 1 -- 172.21.15.7:0/3174506474 shutdown_connections 2024-01-21T03:20:07.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.094+0000 7f83c76a9640 1 -- 172.21.15.7:0/3174506474 wait complete. 2024-01-21T03:20:07.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.095+0000 7f83c76a9640 10 monclient: build_initial_monmap 2024-01-21T03:20:07.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.095+0000 7f83c76a9640 10 monclient: monmap: 2024-01-21T03:20:07.083 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:07.084 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:07.084 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:07.084 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:07.084 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:07.084 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.084 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:07.084 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:07.084 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:07.084 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.095+0000 7f83c76a9640 1 librados: starting msgr at 2024-01-21T03:20:07.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.095+0000 7f83c76a9640 1 librados: starting objecter 2024-01-21T03:20:07.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.095+0000 7f83c76a9640 1 Processor -- start 2024-01-21T03:20:07.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.095+0000 7f330c505640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T03:20:07.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.095+0000 7f330c505640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T03:20:07.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.095+0000 7f330c505640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:07.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.095+0000 7f330c505640 1 -- 172.21.15.7:0/2087811721 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f33041d76a0 con 0x7f33040b2210 2024-01-21T03:20:07.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.095+0000 7f83c76a9640 1 -- start start 2024-01-21T03:20:07.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.095+0000 7f83c76a9640 1 librados: setting wanted keys 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.095+0000 7f83c76a9640 1 librados: calling monclient init 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.095+0000 7f83c76a9640 10 monclient: init 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83c00a8a40 0x7f83c00ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 10 monclient(hunting): picked mon.noname-a con 0x7f83c00a8a40 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f83c00a8e20 0x7f83c00abd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 10 monclient(hunting): picked mon.noname-c con 0x7f83c00a8e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83c00b2210 0x7f83c0137c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 10 monclient(hunting): picked mon.noname-b con 0x7f83c00b2210 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c00ad820 con 0x7f83c00a8a40 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c00ac290 con 0x7f83c00b2210 2024-01-21T03:20:07.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:07.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c00ac500 con 0x7f83c00a8e20 2024-01-21T03:20:07.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:07.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.096+0000 7f83c76a9640 10 monclient(hunting): authenticate will time out at 1353.412533s 2024-01-21T03:20:07.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.097+0000 7f83c4c44640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f83c00a8e20 0x7f83c00abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:07.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.097+0000 7f83c5c46640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83c00b2210 0x7f83c0137c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:07.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.097+0000 7f83c5c46640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83c00b2210 0x7f83c0137c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:56754/0 (socket says 172.21.15.7:56754) 2024-01-21T03:20:07.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.097+0000 7f83c5c46640 1 -- 172.21.15.7:0/1686484046 learned_addr learned my addr 172.21.15.7:0/1686484046 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:07.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.097+0000 7f83c5c46640 10 monclient(hunting): get_auth_request con 0x7f83c00b2210 auth_method 0 2024-01-21T03:20:07.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.097+0000 7f83c5c46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:07.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.097+0000 7f83c5c46640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:07.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.097+0000 7f83c5c46640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:07.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.098+0000 7f83c4c44640 10 monclient(hunting): get_auth_request con 0x7f83c00a8e20 auth_method 0 2024-01-21T03:20:07.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.098+0000 7f83c4c44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:07.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.098+0000 7f83c4c44640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:07.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.098+0000 7f83c4c44640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:07.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.098+0000 7f83c5c46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:07.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.098+0000 7f83c5c46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:07.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.098+0000 7f83c5c46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:07.090 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.3 2024-01-21T03:20:07.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.101+0000 7f83c5c46640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2024-01-21T03:20:07.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.101+0000 7f83c5c46640 1 -- 172.21.15.7:0/1686484046 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f83c00a8e20 msgr2=0x7f83c00abd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.101+0000 7f83c5c46640 1 --2- 172.21.15.7:0/1686484046 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f83c00a8e20 0x7f83c00abd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.101+0000 7f83c5c46640 1 -- 172.21.15.7:0/1686484046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83c00a8a40 msgr2=0x7f83c00ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.101+0000 7f83c5c46640 1 --2- 172.21.15.7:0/1686484046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83c00a8a40 0x7f83c00ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.101+0000 7f83c5c46640 10 monclient: _finish_hunting 0 2024-01-21T03:20:07.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.101+0000 7f83c5c46640 1 monclient: found mon.noname-b 2024-01-21T03:20:07.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.101+0000 7f83c5c46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:07.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.101+0000 7f83c5c46640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:07.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.101+0000 7f83c5c46640 1 -- 172.21.15.7:0/1686484046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc009400 con 0x7f83c00b2210 2024-01-21T03:20:07.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.102+0000 7f32faffd640 1 -- 172.21.15.7:0/2087811721 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f32fc017020 con 0x7f33040b2210 2024-01-21T03:20:07.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.102+0000 7f32faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T03:20:07.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.102+0000 7f32faffd640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:07.091 INFO:teuthology.orchestra.run.smithi007.stdout:60129542151 2024-01-21T03:20:07.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.103+0000 7f83c5c46640 1 --2- 172.21.15.7:0/1686484046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83c00b2210 0x7f83c0137c90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f83b800c780 tx=0x7f83b800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:07.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.103+0000 7f83b5ffb640 10 client.?.objecter ms_handle_connect 0x7f83c00b2210 2024-01-21T03:20:07.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.103+0000 7f83b5ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:07.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 10 librados: watch_flush enter 2024-01-21T03:20:07.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 10 librados: watch_flush exit 2024-01-21T03:20:07.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 20 client.4310.objecter shutdown clearing up homeless session... 2024-01-21T03:20:07.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 10 client.4310.objecter successfully canceled tick 2024-01-21T03:20:07.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 1 -- 172.21.15.7:0/2087811721 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f32e806c8b0 msgr2=0x7f32e806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 1 --2- 172.21.15.7:0/2087811721 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f32e806c8b0 0x7f32e806ed70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3300001560 tx=0x7f3300007990 comp rx=0 tx=0).stop 2024-01-21T03:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 10 monclient: shutdown 2024-01-21T03:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 1 -- 172.21.15.7:0/2087811721 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 msgr2=0x7f3304137fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 1 --2- 172.21.15.7:0/2087811721 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 0x7f3304137fa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f32fc00c7b0 tx=0x7f32fc00cc70 comp rx=0 tx=0).stop 2024-01-21T03:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 1 -- 172.21.15.7:0/2087811721 shutdown_connections 2024-01-21T03:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 1 --2- 172.21.15.7:0/2087811721 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f32e806c8b0 0x7f32e806ed70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 1 --2- 172.21.15.7:0/2087811721 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f33040b2210 0x7f3304137fa0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 1 --2- 172.21.15.7:0/2087811721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33040a8e20 0x7f3304137a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 1 --2- 172.21.15.7:0/2087811721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33040a8a40 0x7f33041cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.104+0000 7f330c505640 1 -- 172.21.15.7:0/2087811721 >> 172.21.15.7:0/2087811721 conn(0x7f3304066f10 msgr2=0x7f3304145b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.105+0000 7f330c505640 1 -- 172.21.15.7:0/2087811721 shutdown_connections 2024-01-21T03:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.105+0000 7f330c505640 1 -- 172.21.15.7:0/2087811721 wait complete. 2024-01-21T03:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.105+0000 7f330c505640 1 librados: shutdown 2024-01-21T03:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.106+0000 7f83b5ffb640 1 -- 172.21.15.7:0/1686484046 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b8003d80 con 0x7f83c00b2210 2024-01-21T03:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.106+0000 7f83b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.106+0000 7f83b5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.106+0000 7f83b5ffb640 10 monclient: dump: 2024-01-21T03:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.106+0000 7f83b5ffb640 10 monclient: _finish_auth 0 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.106+0000 7f83b5ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.106+0000 7f83b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.106+0000 7f83b5ffb640 1 -- 172.21.15.7:0/1686484046 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b800ce30 con 0x7f83c00b2210 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.106+0000 7f83b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.106+0000 7f83b77fe640 4 set_mon_vals no callback set 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.107+0000 7f83b5ffb640 1 -- 172.21.15.7:0/1686484046 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b80052b0 con 0x7f83c00b2210 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.107+0000 7f83b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.107+0000 7f83b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.107+0000 7f83b5ffb640 10 monclient: dump: 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.107+0000 7f83c76a9640 5 monclient: authenticate success, global_id 4372 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.107+0000 7f83c76a9640 10 monclient: _renew_subs 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.107+0000 7f83c76a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.107+0000 7f83c76a9640 1 -- 172.21.15.7:0/1686484046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c00b16f0 con 0x7f83c00b2210 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.108+0000 7f83c76a9640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.108+0000 7f83c76a9640 10 monclient: _renew_subs 2024-01-21T03:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.108+0000 7f83c76a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.108+0000 7f83c76a9640 1 -- 172.21.15.7:0/1686484046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c01383b0 con 0x7f83c00b2210 2024-01-21T03:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.108+0000 7f83c76a9640 1 librados: init done 2024-01-21T03:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.109+0000 7f83c76a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.109+0000 7f83c76a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.109+0000 7f83c76a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.109+0000 7f83c76a9640 1 -- 172.21.15.7:0/1686484046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c00aa1c0 con 0x7f83c00b2210 2024-01-21T03:20:07.105 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.4 2024-01-21T03:20:07.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.127+0000 7f83b5ffb640 1 -- 172.21.15.7:0/1686484046 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f83b8006b10 con 0x7f83c00b2210 2024-01-21T03:20:07.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.127+0000 7f83b5ffb640 1 --2- 172.21.15.7:0/1686484046 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f83a406c860 0x7f83a406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:07.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.127+0000 7f83c5445640 1 --2- 172.21.15.7:0/1686484046 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f83a406c860 0x7f83a406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:07.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.127+0000 7f83c5445640 10 monclient: get_auth_request con 0x7f83a406c860 auth_method 0 2024-01-21T03:20:07.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f83b5ffb640 10 client.4372.objecter ms_handle_connect 0x7f83a406c860 2024-01-21T03:20:07.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f83c5445640 1 --2- 172.21.15.7:0/1686484046 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f83a406c860 0x7f83a406ed20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f83bc036000 tx=0x7f83bc030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:07.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e777fe640 1 -- 172.21.15.7:0/2552683999 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7002cc60 con 0x7f4e78145af0 2024-01-21T03:20:07.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:07.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:07.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e777fe640 10 monclient: dump: 2024-01-21T03:20:07.115 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:07.115 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:07.115 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.115 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e777fe640 10 monclient: _finish_auth 0 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e777fe640 10 monclient: _check_auth_tickets 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e777fe640 1 -- 172.21.15.7:0/2552683999 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7002cdc0 con 0x7f4e78145af0 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e7ebb0640 5 monclient: authenticate success, global_id 4332 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e7ebb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e777fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e7ebb0640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e777fe640 1 -- 172.21.15.7:0/2552683999 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70034660 con 0x7f4e78145af0 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e7ebb0640 4 set_mon_vals no callback set 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.128+0000 7f4e777fe640 10 monclient: dump: 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:07.117 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f4e7ebb0640 10 monclient: shutdown 2024-01-21T03:20:07.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f4e7ebb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:07.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2552683999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e78145af0 msgr2=0x7f4e7814d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f4e7ebb0640 1 --2- 172.21.15.7:0/2552683999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e78145af0 0x7f4e7814d780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009d90 tx=0x7f4e7002beb0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2552683999 shutdown_connections 2024-01-21T03:20:07.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f4e7ebb0640 1 --2- 172.21.15.7:0/2552683999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e78145af0 0x7f4e7814d780 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2552683999 >> 172.21.15.7:0/2552683999 conn(0x7f4e780b4d60 msgr2=0x7f4e780b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:07.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2552683999 shutdown_connections 2024-01-21T03:20:07.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2552683999 wait complete. 2024-01-21T03:20:07.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f9dc3fff640 1 -- 172.21.15.7:0/3466940951 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9dd0020070 con 0x7f9ddc0a8e20 2024-01-21T03:20:07.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f9dc3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T03:20:07.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f9dc3fff640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:07.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f83b5ffb640 1 -- 172.21.15.7:0/1686484046 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f83b8057ab0 con 0x7f83c00b2210 2024-01-21T03:20:07.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.129+0000 7f83b5ffb640 10 client.4372.objecter ms_dispatch 0x7f83c0000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:07.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.130+0000 7f83b5ffb640 3 client.4372.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:07.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.130+0000 7f83b5ffb640 3 client.4372.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:07.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.130+0000 7f83b5ffb640 20 client.4372.objecter dump_active .. 0 homeless 2024-01-21T03:20:07.120 INFO:teuthology.orchestra.run.smithi007.stdout:90194313224 2024-01-21T03:20:07.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.130+0000 7f4e7ebb0640 10 monclient: build_initial_monmap 2024-01-21T03:20:07.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.130+0000 7f4e7ebb0640 10 monclient: monmap: 2024-01-21T03:20:07.121 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:07.121 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:07.121 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:07.121 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:07.121 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:07.121 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.121 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:07.121 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:07.121 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.130+0000 7f4e7ebb0640 1 librados: starting msgr at 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.130+0000 7f4e7ebb0640 1 librados: starting objecter 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.130+0000 7f4e7ebb0640 1 Processor -- start 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.130+0000 7f4e7ebb0640 1 -- start start 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.130+0000 7f4e7ebb0640 1 librados: setting wanted keys 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.130+0000 7f4e7ebb0640 1 librados: calling monclient init 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.130+0000 7f4e7ebb0640 10 monclient: init 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.131+0000 7f4e7ebb0640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.131+0000 7f4e7ebb0640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.131+0000 7f4e7ebb0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e780b1bb0 0x7f4e78144090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f4e7ebb0640 10 monclient(hunting): picked mon.noname-a con 0x7f4e780b1bb0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f4e7ebb0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e7813f5c0 0x7f4e781445d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f4e7ebb0640 10 monclient(hunting): picked mon.noname-b con 0x7f4e7813f5c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f4e7ebb0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4e78145af0 0x7f4e781d7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f4e7ebb0640 10 monclient(hunting): picked mon.noname-c con 0x7f4e78145af0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f4e7ebb0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:07.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f4e7ebb0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e78143b70 con 0x7f4e780b1bb0 2024-01-21T03:20:07.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f4e7ebb0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:07.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f4e7ebb0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781437b0 con 0x7f4e7813f5c0 2024-01-21T03:20:07.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f4e7ebb0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:07.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f4e7ebb0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e78143970 con 0x7f4e78145af0 2024-01-21T03:20:07.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f4e7ebb0640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:07.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f4e7ebb0640 10 monclient(hunting): authenticate will time out at 1353.448024s 2024-01-21T03:20:07.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.132+0000 7f9de1cd7640 10 librados: watch_flush enter 2024-01-21T03:20:07.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 10 librados: watch_flush exit 2024-01-21T03:20:07.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 20 client.4320.objecter shutdown clearing up homeless session... 2024-01-21T03:20:07.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 10 client.4320.objecter successfully canceled tick 2024-01-21T03:20:07.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f9dc806c8e0 msgr2=0x7f9dc806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 1 --2- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f9dc806c8e0 0x7f9dc806eda0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc009710 tx=0x7f9dcc006c90 comp rx=0 tx=0).stop 2024-01-21T03:20:07.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 10 monclient: shutdown 2024-01-21T03:20:07.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:07.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ddc0a8e20 msgr2=0x7f9ddc140050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 1 --2- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ddc0a8e20 0x7f9ddc140050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c390 tx=0x7f9dd000c850 comp rx=0 tx=0).stop 2024-01-21T03:20:07.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3466940951 shutdown_connections 2024-01-21T03:20:07.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 1 --2- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f9dc806c8e0 0x7f9dc806eda0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 1 --2- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ddc0b2210 0x7f9ddc133d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 1 --2- 172.21.15.7:0/3466940951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ddc0a8e20 0x7f9ddc140050 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 1 --2- 172.21.15.7:0/3466940951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ddc0a8a40 0x7f9ddc144fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3466940951 >> 172.21.15.7:0/3466940951 conn(0x7f9ddc066f10 msgr2=0x7f9ddc13b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:07.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f4e7d14d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4e78145af0 0x7f4e781d7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:07.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3466940951 shutdown_connections 2024-01-21T03:20:07.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 1 -- 172.21.15.7:0/3466940951 wait complete. 2024-01-21T03:20:07.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f4e7d14d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4e78145af0 0x7f4e781d7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60078/0 (socket says 172.21.15.7:60078) 2024-01-21T03:20:07.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f4e7d14d640 1 -- 172.21.15.7:0/2608854045 learned_addr learned my addr 172.21.15.7:0/2608854045 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:07.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f9de1cd7640 1 librados: shutdown 2024-01-21T03:20:07.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f4e7d14d640 10 monclient(hunting): get_auth_request con 0x7f4e78145af0 auth_method 0 2024-01-21T03:20:07.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f4e7d14d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:07.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f4e7d14d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:07.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.133+0000 7f4e7d14d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:07.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.138+0000 7f4e7d14d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:07.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.138+0000 7f4e7d14d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:07.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.138+0000 7f4e7d14d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:07.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313224 for osd.0 2024-01-21T03:20:07.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.146+0000 7f4e7d14d640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-01-21T03:20:07.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.146+0000 7f4e7d14d640 1 -- 172.21.15.7:0/2608854045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e7813f5c0 msgr2=0x7f4e781445d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.146+0000 7f4e7d14d640 1 --2- 172.21.15.7:0/2608854045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e7813f5c0 0x7f4e781445d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.146+0000 7f4e7d14d640 1 -- 172.21.15.7:0/2608854045 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e780b1bb0 msgr2=0x7f4e78144090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.146+0000 7f4e7d14d640 1 --2- 172.21.15.7:0/2608854045 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e780b1bb0 0x7f4e78144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.146+0000 7f4e7d14d640 10 monclient: _finish_hunting 0 2024-01-21T03:20:07.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.146+0000 7f4e7d14d640 1 monclient: found mon.noname-c 2024-01-21T03:20:07.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.146+0000 7f4e7d14d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:07.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.146+0000 7f4e7d14d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:20:07.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.146+0000 7f4e7d14d640 1 -- 172.21.15.7:0/2608854045 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e70009a70 con 0x7f4e78145af0 2024-01-21T03:20:07.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.150+0000 7f4e7d14d640 1 --2- 172.21.15.7:0/2608854045 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4e78145af0 0x7f4e781d7670 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009d90 tx=0x7f4e70004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:07.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.150+0000 7f4e757fa640 10 client.?.objecter ms_handle_connect 0x7f4e78145af0 2024-01-21T03:20:07.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.150+0000 7f4e757fa640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:07.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e757fa640 1 -- 172.21.15.7:0/2608854045 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7003e610 con 0x7f4e78145af0 2024-01-21T03:20:07.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:07.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e757fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:07.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e757fa640 10 monclient: dump: 2024-01-21T03:20:07.229 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:07.229 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:07.229 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.229 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.229 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:07.229 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e757fa640 10 monclient: _finish_auth 0 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e757fa640 10 monclient: _check_auth_tickets 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e757fa640 1 -- 172.21.15.7:0/2608854045 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7003e790 con 0x7f4e78145af0 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e7ebb0640 5 monclient: authenticate success, global_id 4316 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e7ebb0640 10 monclient: _renew_subs 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e7ebb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2608854045 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e78144b10 con 0x7f4e78145af0 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e757fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e757fa640 1 -- 172.21.15.7:0/2608854045 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7003eb40 con 0x7f4e78145af0 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e757fa640 10 monclient: dump: 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e76ffd640 4 set_mon_vals no callback set 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e7ebb0640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e7ebb0640 10 monclient: _renew_subs 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.241+0000 7f4e7ebb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.242+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2608854045 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e781d7d90 con 0x7f4e78145af0 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.242+0000 7f4e7ebb0640 1 librados: init done 2024-01-21T03:20:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.243+0000 7f4e7ebb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:07.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.243+0000 7f4e7ebb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:07.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.243+0000 7f4e7ebb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:07.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.243+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2608854045 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e780acf10 con 0x7f4e78145af0 2024-01-21T03:20:07.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.315+0000 7ff327661640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T03:20:07.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.315+0000 7ff327661640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T03:20:07.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.315+0000 7ff327661640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:07.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.315+0000 7ff327661640 1 -- 172.21.15.7:0/20091628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff3200b2ed0 con 0x7ff32006fe10 2024-01-21T03:20:07.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.317+0000 7fde515d0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T03:20:07.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.317+0000 7fde515d0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T03:20:07.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.317+0000 7fde515d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:07.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.317+0000 7fde515d0640 1 -- 172.21.15.7:0/1836678464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fde4c142e60 con 0x7fde4c0a9d10 2024-01-21T03:20:07.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.324+0000 7fde337fe640 1 -- 172.21.15.7:0/1836678464 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fde4c142e60 con 0x7fde4c0a9d10 2024-01-21T03:20:07.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.324+0000 7fde337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T03:20:07.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.324+0000 7fde337fe640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:07.310 INFO:teuthology.orchestra.run.smithi007.stdout:77309411337 2024-01-21T03:20:07.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 10 librados: watch_flush enter 2024-01-21T03:20:07.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 10 librados: watch_flush exit 2024-01-21T03:20:07.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 20 client.4329.objecter shutdown clearing up homeless session... 2024-01-21T03:20:07.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 10 client.4329.objecter successfully canceled tick 2024-01-21T03:20:07.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 1 -- 172.21.15.7:0/1836678464 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fde3806c8e0 msgr2=0x7fde3806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 1 --2- 172.21.15.7:0/1836678464 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fde3806c8e0 0x7fde3806eda0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde3c000f60 tx=0x7fde3c007a80 comp rx=0 tx=0).stop 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 10 monclient: shutdown 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 1 -- 172.21.15.7:0/1836678464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c0a9d10 msgr2=0x7fde4c13fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 1 --2- 172.21.15.7:0/1836678464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c0a9d10 0x7fde4c13fd60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fde400098c0 tx=0x7fde400046c0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 1 -- 172.21.15.7:0/1836678464 shutdown_connections 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 1 --2- 172.21.15.7:0/1836678464 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fde3806c8e0 0x7fde3806eda0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 1 --2- 172.21.15.7:0/1836678464 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fde4c0b2310 0x7fde4c1daa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 1 --2- 172.21.15.7:0/1836678464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c0a9d10 0x7fde4c13fd60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 1 --2- 172.21.15.7:0/1836678464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde4c0a8b20 0x7fde4c136fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 1 -- 172.21.15.7:0/1836678464 >> 172.21.15.7:0/1836678464 conn(0x7fde4c066fb0 msgr2=0x7fde4c133960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 1 -- 172.21.15.7:0/1836678464 shutdown_connections 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 1 -- 172.21.15.7:0/1836678464 wait complete. 2024-01-21T03:20:07.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.326+0000 7fde515d0640 1 librados: shutdown 2024-01-21T03:20:07.322 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411338 got 77309411337 for osd.5 2024-01-21T03:20:07.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.365+0000 7f83b5ffb640 1 -- 172.21.15.7:0/1686484046 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f83b8053df0 con 0x7f83c00b2210 2024-01-21T03:20:07.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.365+0000 7f83b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:07.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.365+0000 7f83b5ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.370+0000 7ff315ffb640 1 -- 172.21.15.7:0/20091628 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff31c057540 con 0x7ff32006fe10 2024-01-21T03:20:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.370+0000 7ff315ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T03:20:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.370+0000 7ff315ffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:07.356 INFO:teuthology.orchestra.run.smithi007.stdout:90194313223 2024-01-21T03:20:07.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 10 librados: watch_flush enter 2024-01-21T03:20:07.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 10 librados: watch_flush exit 2024-01-21T03:20:07.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 20 client.4366.objecter shutdown clearing up homeless session... 2024-01-21T03:20:07.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 10 client.4366.objecter successfully canceled tick 2024-01-21T03:20:07.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 1 -- 172.21.15.7:0/20091628 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ff304070cf0 msgr2=0x7ff3040731b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 1 --2- 172.21.15.7:0/20091628 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ff304070cf0 0x7ff3040731b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff310036000 tx=0x7ff310030000 comp rx=0 tx=0).stop 2024-01-21T03:20:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 10 monclient: shutdown 2024-01-21T03:20:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 1 -- 172.21.15.7:0/20091628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff32006fe10 msgr2=0x7ff32013df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 1 --2- 172.21.15.7:0/20091628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff32006fe10 0x7ff32013df20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff31c00c4a0 tx=0x7ff31c00c960 comp rx=0 tx=0).stop 2024-01-21T03:20:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 1 -- 172.21.15.7:0/20091628 shutdown_connections 2024-01-21T03:20:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 1 --2- 172.21.15.7:0/20091628 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7ff304070cf0 0x7ff3040731b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 1 --2- 172.21.15.7:0/20091628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff32013f160 0x7ff3200adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 1 --2- 172.21.15.7:0/20091628 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff32013c300 0x7ff3201d68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 1 --2- 172.21.15.7:0/20091628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff32006fe10 0x7ff32013df20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 1 -- 172.21.15.7:0/20091628 >> 172.21.15.7:0/20091628 conn(0x7ff3200b4910 msgr2=0x7ff3200b3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 1 -- 172.21.15.7:0/20091628 shutdown_connections 2024-01-21T03:20:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 1 -- 172.21.15.7:0/20091628 wait complete. 2024-01-21T03:20:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.372+0000 7ff327661640 1 librados: shutdown 2024-01-21T03:20:07.369 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313224 got 90194313223 for osd.1 2024-01-21T03:20:07.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.529+0000 7f4e757fa640 1 -- 172.21.15.7:0/2608854045 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4e7003b050 con 0x7f4e78145af0 2024-01-21T03:20:07.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.530+0000 7f4e757fa640 1 --2- 172.21.15.7:0/2608854045 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4e5c06c890 0x7f4e5c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:07.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.530+0000 7f4e7c94c640 1 --2- 172.21.15.7:0/2608854045 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4e5c06c890 0x7f4e5c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:07.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.530+0000 7f4e7c94c640 10 monclient: get_auth_request con 0x7f4e5c06c890 auth_method 0 2024-01-21T03:20:07.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.530+0000 7f4e757fa640 10 client.4316.objecter ms_handle_connect 0x7f4e5c06c890 2024-01-21T03:20:07.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.530+0000 7f4e7c94c640 1 --2- 172.21.15.7:0/2608854045 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4e5c06c890 0x7f4e5c06ed50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c001560 tx=0x7f4e6c007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:07.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.545+0000 7f83c76a9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-21T03:20:07.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.545+0000 7f83c76a9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-21T03:20:07.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.545+0000 7f83c76a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:07.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.545+0000 7f83c76a9640 1 -- 172.21.15.7:0/1686484046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f83c004e970 con 0x7f83c00b2210 2024-01-21T03:20:07.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.549+0000 7f83b5ffb640 1 -- 172.21.15.7:0/1686484046 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f83b8017070 con 0x7f83c00b2210 2024-01-21T03:20:07.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.549+0000 7f83b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-21T03:20:07.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.549+0000 7f83b5ffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:07.535 INFO:teuthology.orchestra.run.smithi007.stdout:90194313224 2024-01-21T03:20:07.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.550+0000 7f4e757fa640 1 -- 172.21.15.7:0/2608854045 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4e7007b750 con 0x7f4e78145af0 2024-01-21T03:20:07.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.550+0000 7f4e757fa640 10 client.4316.objecter ms_dispatch 0x7f4e78000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:07.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.550+0000 7f4e757fa640 3 client.4316.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:07.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.550+0000 7f4e757fa640 3 client.4316.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:07.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.550+0000 7f4e757fa640 20 client.4316.objecter dump_active .. 0 homeless 2024-01-21T03:20:07.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.550+0000 7f4e757fa640 1 -- 172.21.15.7:0/2608854045 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4e70077390 con 0x7f4e78145af0 2024-01-21T03:20:07.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.550+0000 7f4e757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:07.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.550+0000 7f4e757fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:07.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.551+0000 7f83c76a9640 10 librados: watch_flush enter 2024-01-21T03:20:07.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 10 librados: watch_flush exit 2024-01-21T03:20:07.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 20 client.4372.objecter shutdown clearing up homeless session... 2024-01-21T03:20:07.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 10 client.4372.objecter successfully canceled tick 2024-01-21T03:20:07.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 1 -- 172.21.15.7:0/1686484046 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f83a406c860 msgr2=0x7f83a406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 1 --2- 172.21.15.7:0/1686484046 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f83a406c860 0x7f83a406ed20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f83bc036000 tx=0x7f83bc030000 comp rx=0 tx=0).stop 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 10 monclient: shutdown 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 1 -- 172.21.15.7:0/1686484046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83c00b2210 msgr2=0x7f83c0137c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 1 --2- 172.21.15.7:0/1686484046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83c00b2210 0x7f83c0137c90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f83b800c780 tx=0x7f83b800cc40 comp rx=0 tx=0).stop 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 1 -- 172.21.15.7:0/1686484046 shutdown_connections 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 1 --2- 172.21.15.7:0/1686484046 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f83a406c860 0x7f83a406ed20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 1 --2- 172.21.15.7:0/1686484046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83c00b2210 0x7f83c0137c90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 1 --2- 172.21.15.7:0/1686484046 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f83c00a8e20 0x7f83c00abd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 1 --2- 172.21.15.7:0/1686484046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83c00a8a40 0x7f83c00ab6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 1 -- 172.21.15.7:0/1686484046 >> 172.21.15.7:0/1686484046 conn(0x7f83c0066f10 msgr2=0x7f83c013b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 1 -- 172.21.15.7:0/1686484046 shutdown_connections 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 1 -- 172.21.15.7:0/1686484046 wait complete. 2024-01-21T03:20:07.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.552+0000 7f83c76a9640 1 librados: shutdown 2024-01-21T03:20:07.552 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313224 got 90194313224 for osd.2 2024-01-21T03:20:07.552 DEBUG:teuthology.parallel:result is None 2024-01-21T03:20:07.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.765+0000 7f4e7ebb0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-21T03:20:07.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.765+0000 7f4e7ebb0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-21T03:20:07.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.765+0000 7f4e7ebb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:07.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.765+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2608854045 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4e7804e970 con 0x7f4e78145af0 2024-01-21T03:20:07.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.769+0000 7f4e757fa640 1 -- 172.21.15.7:0/2608854045 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4e7004e090 con 0x7f4e78145af0 2024-01-21T03:20:07.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.769+0000 7f4e757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-21T03:20:07.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.769+0000 7f4e757fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:07.755 INFO:teuthology.orchestra.run.smithi007.stdout:60129542152 2024-01-21T03:20:07.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 10 librados: watch_flush enter 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 10 librados: watch_flush exit 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 20 client.4316.objecter shutdown clearing up homeless session... 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 10 client.4316.objecter successfully canceled tick 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2608854045 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4e5c06c890 msgr2=0x7f4e5c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 1 --2- 172.21.15.7:0/2608854045 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4e5c06c890 0x7f4e5c06ed50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c001560 tx=0x7f4e6c007990 comp rx=0 tx=0).stop 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 10 monclient: shutdown 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2608854045 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4e78145af0 msgr2=0x7f4e781d7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 1 --2- 172.21.15.7:0/2608854045 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4e78145af0 0x7f4e781d7670 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009d90 tx=0x7f4e70004a40 comp rx=0 tx=0).stop 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2608854045 shutdown_connections 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 1 --2- 172.21.15.7:0/2608854045 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4e5c06c890 0x7f4e5c06ed50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 1 --2- 172.21.15.7:0/2608854045 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4e78145af0 0x7f4e781d7670 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 1 --2- 172.21.15.7:0/2608854045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e7813f5c0 0x7f4e781445d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 1 --2- 172.21.15.7:0/2608854045 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e780b1bb0 0x7f4e78144090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2608854045 >> 172.21.15.7:0/2608854045 conn(0x7f4e780b4d60 msgr2=0x7f4e780b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:07.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2608854045 shutdown_connections 2024-01-21T03:20:07.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 1 -- 172.21.15.7:0/2608854045 wait complete. 2024-01-21T03:20:07.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.771+0000 7f4e7ebb0640 1 librados: shutdown 2024-01-21T03:20:07.768 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.7 2024-01-21T03:20:07.769 DEBUG:teuthology.parallel:result is None 2024-01-21T03:20:07.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-21T03:20:07.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.918+0000 7f2b7c76c640 10 monclient: get_monmap_and_config 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.918+0000 7f2b7c76c640 10 monclient: build_initial_monmap 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.919+0000 7f2b7c76c640 10 monclient: monmap: 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:07.919954+0000 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:07.919954+0000 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:07.905 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.919+0000 7f2b7c76c640 1 Processor -- start 2024-01-21T03:20:07.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.919+0000 7f2b7c76c640 1 -- start start 2024-01-21T03:20:07.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.919+0000 7f2b7c76c640 10 monclient: init 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.920+0000 7f2b7c76c640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.920+0000 7f2b7c76c640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.920+0000 7f2b7c76c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2b740a8ab0 0x7f2b74142640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.920+0000 7f2b7c76c640 10 monclient(hunting): picked mon.noname-c con 0x7f2b740a8ab0 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.920+0000 7f2b7c76c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b740b2310 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.920+0000 7f2b7c76c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b740a8e90 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.920+0000 7f2b7c76c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b740aa7d0 con 0x7f2b740b2310 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.920+0000 7f2b7c76c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b740aaa20 con 0x7f2b740a8e90 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.920+0000 7f2b7c76c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.920+0000 7f2b7c76c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b740aad00 con 0x7f2b740a8ab0 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.920+0000 7f2b7c76c640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.920+0000 7f2b7c76c640 10 monclient(hunting): authenticate will time out at 1354.236177s 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.921+0000 7f2b79d07640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2b740b2310 0x7f2b7413f130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33754/0 (socket says 172.21.15.7:33754) 2024-01-21T03:20:07.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.921+0000 7f2b79d07640 1 -- 172.21.15.7:0/2905385384 learned_addr learned my addr 172.21.15.7:0/2905385384 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:07.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.921+0000 7f2b7a508640 1 --2- 172.21.15.7:0/2905385384 >> v2:172.21.15.177:3300/0 conn(0x7f2b740a8ab0 0x7f2b74142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:07.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.922+0000 7f2b7a508640 10 monclient(hunting): get_auth_request con 0x7f2b740a8ab0 auth_method 0 2024-01-21T03:20:07.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.922+0000 7f2b7a508640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:07.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.922+0000 7f2b7a508640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:07.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.922+0000 7f2b7a508640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:07.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.922+0000 7f2b7a508640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:07.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.922+0000 7f2b7a508640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:07.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.922+0000 7f2b7a508640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:07.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.923+0000 7f2b79506640 1 -- 172.21.15.7:0/2905385384 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558402668 0 0) 0x7f2b740aa7d0 con 0x7f2b740b2310 2024-01-21T03:20:07.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.923+0000 7f2b79506640 10 monclient(hunting): _negotiate 2024-01-21T03:20:07.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.923+0000 7f2b79506640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:07.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.923+0000 7f2b79506640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:07.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.923+0000 7f2b79506640 10 monclient(hunting): my global_id is 4338 2024-01-21T03:20:07.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.923+0000 7f2b79506640 1 -- 172.21.15.7:0/2905385384 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b60003520 con 0x7f2b740b2310 2024-01-21T03:20:07.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.924+0000 7f2b7a508640 10 monclient(hunting): handle_auth_done global_id 4375 payload 274 2024-01-21T03:20:07.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.924+0000 7f2b7a508640 1 -- 172.21.15.7:0/2905385384 >> v1:172.21.15.177:6789/0 conn(0x7f2b740a8e90 legacy=0x7f2b7414c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.924+0000 7f2b7a508640 1 -- 172.21.15.7:0/2905385384 >> v1:172.21.15.7:6789/0 conn(0x7f2b740b2310 legacy=0x7f2b7413f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.924+0000 7f2b7a508640 10 monclient: _finish_hunting 0 2024-01-21T03:20:07.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.924+0000 7f2b7a508640 1 monclient: found mon.noname-c 2024-01-21T03:20:07.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.924+0000 7f2b7a508640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:07.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.924+0000 7f2b7a508640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:20:07.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.924+0000 7f2b7a508640 1 -- 172.21.15.7:0/2905385384 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b740aafe0 con 0x7f2b740a8ab0 2024-01-21T03:20:07.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.929+0000 7f2b7a508640 1 --2- 172.21.15.7:0/2905385384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b740a8ab0 0x7f2b74142640 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0097a0 tx=0x7f2b6c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=207df9565b01b97c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:07.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b79506640 1 -- 172.21.15.7:0/2905385384 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c030610 con 0x7f2b740a8ab0 2024-01-21T03:20:07.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b79506640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b79506640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b79506640 10 monclient: dump: 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b79506640 10 monclient: _finish_auth 0 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b79506640 10 monclient: _check_auth_tickets 2024-01-21T03:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b79506640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b79506640 1 -- 172.21.15.7:0/2905385384 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6c030770 con 0x7f2b740a8ab0 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b79506640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b79506640 1 -- 172.21.15.7:0/2905385384 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c030b00 con 0x7f2b740a8ab0 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b79506640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b79506640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b79506640 10 monclient: dump: 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:07.921 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:07.922 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b7c76c640 5 monclient: authenticate success, global_id 4375 2024-01-21T03:20:07.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b7c76c640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:07.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b7c76c640 4 set_mon_vals no callback set 2024-01-21T03:20:07.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b7c76c640 10 monclient: shutdown 2024-01-21T03:20:07.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b7c76c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:07.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b7c76c640 1 -- 172.21.15.7:0/2905385384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b740a8ab0 msgr2=0x7f2b74142640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.933+0000 7f2b7c76c640 1 --2- 172.21.15.7:0/2905385384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b740a8ab0 0x7f2b74142640 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0097a0 tx=0x7f2b6c02b960 comp rx=0 tx=0).stop 2024-01-21T03:20:07.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.934+0000 7f2b7c76c640 1 -- 172.21.15.7:0/2905385384 shutdown_connections 2024-01-21T03:20:07.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.934+0000 7f2b7c76c640 1 --2- 172.21.15.7:0/2905385384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b740a8ab0 0x7f2b74142640 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.934+0000 7f2b7c76c640 1 -- 172.21.15.7:0/2905385384 >> 172.21.15.7:0/2905385384 conn(0x7f2b74066f60 msgr2=0x7f2b740b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:07.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.934+0000 7f2b7c76c640 1 -- 172.21.15.7:0/2905385384 shutdown_connections 2024-01-21T03:20:07.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.934+0000 7f2b7c76c640 1 -- 172.21.15.7:0/2905385384 wait complete. 2024-01-21T03:20:07.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.934+0000 7f2b7c76c640 10 monclient: build_initial_monmap 2024-01-21T03:20:07.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.934+0000 7f2b7c76c640 10 monclient: monmap: 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 1 librados: starting msgr at 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 1 librados: starting objecter 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 1 Processor -- start 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 1 -- start start 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 1 librados: setting wanted keys 2024-01-21T03:20:07.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 1 librados: calling monclient init 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 10 monclient: init 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b740a8ab0 0x7f2b740b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 10 monclient(hunting): picked mon.noname-a con 0x7f2b740a8ab0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2b740a8e90 0x7f2b740aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 10 monclient(hunting): picked mon.noname-c con 0x7f2b740a8e90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b740b2310 0x7f2b740af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 10 monclient(hunting): picked mon.noname-b con 0x7f2b740b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b740b01b0 con 0x7f2b740a8ab0 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b740b08b0 con 0x7f2b740b2310 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b740b0a70 con 0x7f2b740a8e90 2024-01-21T03:20:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.935+0000 7f2b7c76c640 10 monclient(hunting): authenticate will time out at 1354.251463s 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7ad09640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b740b2310 0x7f2b740af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7a508640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b740a8ab0 0x7f2b740b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b79d07640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2b740a8e90 0x7f2b740aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7ad09640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b740b2310 0x7f2b740af080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:56794/0 (socket says 172.21.15.7:56794) 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7ad09640 1 -- 172.21.15.7:0/113972815 learned_addr learned my addr 172.21.15.7:0/113972815 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7ad09640 10 monclient(hunting): get_auth_request con 0x7f2b740b2310 auth_method 0 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7ad09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7ad09640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7ad09640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b79d07640 10 monclient(hunting): get_auth_request con 0x7f2b740a8e90 auth_method 0 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b79d07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b79d07640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:07.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b79d07640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7a508640 10 monclient(hunting): get_auth_request con 0x7f2b740a8ab0 auth_method 0 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7a508640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7a508640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7a508640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7ad09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7ad09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.936+0000 7f2b7ad09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.937+0000 7f2b79d07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.937+0000 7f2b79d07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.937+0000 7f2b79d07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.937+0000 7f2b7a508640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.937+0000 7f2b7a508640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.937+0000 7f2b7a508640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.939+0000 7f2b7ad09640 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.939+0000 7f2b7ad09640 1 -- 172.21.15.7:0/113972815 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2b740a8e90 msgr2=0x7f2b740aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.939+0000 7f2b7ad09640 1 --2- 172.21.15.7:0/113972815 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2b740a8e90 0x7f2b740aeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.939+0000 7f2b7ad09640 1 -- 172.21.15.7:0/113972815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b740a8ab0 msgr2=0x7f2b740b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:07.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.939+0000 7f2b7ad09640 1 --2- 172.21.15.7:0/113972815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b740a8ab0 0x7f2b740b0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:07.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.939+0000 7f2b7ad09640 10 monclient: _finish_hunting 0 2024-01-21T03:20:07.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.939+0000 7f2b7ad09640 1 monclient: found mon.noname-b 2024-01-21T03:20:07.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.939+0000 7f2b7ad09640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:07.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.939+0000 7f2b7ad09640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:07.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.939+0000 7f2b7ad09640 1 -- 172.21.15.7:0/113972815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6c009400 con 0x7f2b740b2310 2024-01-21T03:20:07.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.941+0000 7f2b7ad09640 1 --2- 172.21.15.7:0/113972815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b740b2310 0x7f2b740af080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000c7a0 tx=0x7f2b7000cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:07.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.941+0000 7f2b6affd640 10 client.?.objecter ms_handle_connect 0x7f2b740b2310 2024-01-21T03:20:07.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.941+0000 7f2b6affd640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:07.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b6affd640 1 -- 172.21.15.7:0/113972815 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70003b70 con 0x7f2b740b2310 2024-01-21T03:20:07.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:07.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b6affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:07.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b6affd640 10 monclient: dump: 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b6affd640 10 monclient: _finish_auth 0 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b6affd640 10 monclient: _check_auth_tickets 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b6affd640 1 -- 172.21.15.7:0/113972815 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b70003cf0 con 0x7f2b740b2310 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b6affd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b6affd640 1 -- 172.21.15.7:0/113972815 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70003ea0 con 0x7f2b740b2310 2024-01-21T03:20:07.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b7c76c640 5 monclient: authenticate success, global_id 4381 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b6affd640 10 monclient: dump: 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b7c76c640 10 monclient: _renew_subs 2024-01-21T03:20:07.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b78d05640 4 set_mon_vals no callback set 2024-01-21T03:20:07.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b7c76c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:07.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b7c76c640 1 -- 172.21.15.7:0/113972815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b740adf50 con 0x7f2b740b2310 2024-01-21T03:20:07.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b7c76c640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:07.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b7c76c640 10 monclient: _renew_subs 2024-01-21T03:20:07.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b7c76c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:07.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b7c76c640 1 -- 172.21.15.7:0/113972815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b740b15f0 con 0x7f2b740b2310 2024-01-21T03:20:07.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.945+0000 7f2b7c76c640 1 librados: init done 2024-01-21T03:20:07.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.946+0000 7f2b7c76c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:07.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.946+0000 7f2b7c76c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:07.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.946+0000 7f2b7c76c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:07.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:07.946+0000 7f2b7c76c640 1 -- 172.21.15.7:0/113972815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b740b0bb0 con 0x7f2b740b2310 2024-01-21T03:20:08.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-21T03:20:08.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-21T03:20:08.131 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-21T03:20:08.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.189+0000 7f2b6affd640 1 -- 172.21.15.7:0/113972815 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2b7000aa90 con 0x7f2b740b2310 2024-01-21T03:20:08.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.190+0000 7f2b6affd640 1 --2- 172.21.15.7:0/113972815 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f2b5806c8b0 0x7f2b5806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.190+0000 7f2b7a508640 1 --2- 172.21.15.7:0/113972815 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f2b5806c8b0 0x7f2b5806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.190+0000 7f2b7a508640 10 monclient: get_auth_request con 0x7f2b5806c8b0 auth_method 0 2024-01-21T03:20:08.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.191+0000 7f2b7a508640 1 --2- 172.21.15.7:0/113972815 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f2b5806c8b0 0x7f2b5806ed70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c036000 tx=0x7f2b6c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.191+0000 7f2b6affd640 10 client.4381.objecter ms_handle_connect 0x7f2b5806c8b0 2024-01-21T03:20:08.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.192+0000 7f2b6affd640 1 -- 172.21.15.7:0/113972815 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2b70058a10 con 0x7f2b740b2310 2024-01-21T03:20:08.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.192+0000 7f2b6affd640 10 client.4381.objecter ms_dispatch 0x7f2b74000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:08.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.192+0000 7f2b6affd640 3 client.4381.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:08.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.192+0000 7f2b6affd640 3 client.4381.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:08.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.192+0000 7f2b6affd640 20 client.4381.objecter dump_active .. 0 homeless 2024-01-21T03:20:08.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.203+0000 7f2b6affd640 1 -- 172.21.15.7:0/113972815 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2b70054d50 con 0x7f2b740b2310 2024-01-21T03:20:08.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.203+0000 7f2b6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.203+0000 7f2b6affd640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.221+0000 7f1d0fd05640 10 monclient: get_monmap_and_config 2024-01-21T03:20:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.221+0000 7f1d0fd05640 10 monclient: build_initial_monmap 2024-01-21T03:20:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.221+0000 7f1d0fd05640 10 monclient: monmap: 2024-01-21T03:20:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:08.222588+0000 2024-01-21T03:20:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:08.222588+0000 2024-01-21T03:20:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.208 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:08.208 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:08.208 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:08.208 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:08.208 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:08.208 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.222+0000 7f1d0fd05640 1 Processor -- start 2024-01-21T03:20:08.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.222+0000 7f1d0fd05640 1 -- start start 2024-01-21T03:20:08.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.222+0000 7f1d0fd05640 10 monclient: init 2024-01-21T03:20:08.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.223+0000 7f1d0fd05640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:08.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.223+0000 7f1d0fd05640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-01-21T03:20:08.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.223+0000 7f1d0fd05640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d080a8b20 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:08.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.223+0000 7f1d0fd05640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1d080a9d10 0x7f1d0813f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.223+0000 7f1d0fd05640 10 monclient(hunting): picked mon.noname-b con 0x7f1d080a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:08.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.223+0000 7f1d0fd05640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d080b2310 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:08.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.223+0000 7f1d0fd05640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d081456f0 con 0x7f1d080a8b20 2024-01-21T03:20:08.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.223+0000 7f1d0fd05640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d080a98c0 con 0x7f1d080b2310 2024-01-21T03:20:08.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.223+0000 7f1d0fd05640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.223+0000 7f1d0fd05640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d08146290 con 0x7f1d080a9d10 2024-01-21T03:20:08.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.223+0000 7f1d0fd05640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:08.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.223+0000 7f1d0fd05640 10 monclient(hunting): authenticate will time out at 1354.539434s 2024-01-21T03:20:08.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.223+0000 7f1d0d2a0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1d080a9d10 0x7f1d0813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.224+0000 7f1d0d2a0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1d080a9d10 0x7f1d0813f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60096/0 (socket says 172.21.15.7:60096) 2024-01-21T03:20:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.224+0000 7f1d0d2a0640 1 -- 172.21.15.7:0/1336940169 learned_addr learned my addr 172.21.15.7:0/1336940169 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.224+0000 7f1d0d2a0640 10 monclient(hunting): get_auth_request con 0x7f1d080a9d10 auth_method 0 2024-01-21T03:20:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.224+0000 7f1d0d2a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.224+0000 7f1d0d2a0640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.224+0000 7f1d0d2a0640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.225+0000 7f1d0d2a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.225+0000 7f1d0d2a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:08.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.225+0000 7f1d0d2a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:08.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.227+0000 7f1d0d2a0640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-01-21T03:20:08.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.227+0000 7f1d0d2a0640 1 -- 172.21.15.7:0/1336940169 >> v1:172.21.15.177:6789/0 conn(0x7f1d080b2310 legacy=0x7f1d0813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:08.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.227+0000 7f1d0d2a0640 1 -- 172.21.15.7:0/1336940169 >> v1:172.21.15.7:6789/0 conn(0x7f1d080a8b20 legacy=0x7f1d08145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:08.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.227+0000 7f1d0d2a0640 10 monclient: _finish_hunting 0 2024-01-21T03:20:08.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.227+0000 7f1d0d2a0640 1 monclient: found mon.noname-b 2024-01-21T03:20:08.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.227+0000 7f1d0d2a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:08.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.227+0000 7f1d0d2a0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:08.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.227+0000 7f1d0d2a0640 1 -- 172.21.15.7:0/1336940169 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d081463d0 con 0x7f1d080a9d10 2024-01-21T03:20:08.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.232+0000 7f1d0d2a0640 1 --2- 172.21.15.7:0/1336940169 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1d080a9d10 0x7f1d0813f080 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80098a0 tx=0x7f1cf802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c4a908cf586a07d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.276+0000 7faedc4e5640 10 monclient: get_monmap_and_config 2024-01-21T03:20:08.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.276+0000 7faedc4e5640 10 monclient: build_initial_monmap 2024-01-21T03:20:08.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.276+0000 7faedc4e5640 10 monclient: monmap: 2024-01-21T03:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:08.277014+0000 2024-01-21T03:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:08.277014+0000 2024-01-21T03:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.276+0000 7faedc4e5640 1 Processor -- start 2024-01-21T03:20:08.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.276+0000 7faedc4e5640 1 -- start start 2024-01-21T03:20:08.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.276+0000 7faedc4e5640 10 monclient: init 2024-01-21T03:20:08.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:08.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-01-21T03:20:08.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7faed40a8b20 0x7faed4145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 10 monclient(hunting): picked mon.noname-c con 0x7faed40a8b20 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faed40a9d10 0x7faed413f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 10 monclient(hunting): picked mon.noname-b con 0x7faed40a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faed40b2310 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed41456f0 con 0x7faed40b2310 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed40a98c0 con 0x7faed40a8b20 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed41460c0 con 0x7faed40a9d10 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.277+0000 7faedc4e5640 10 monclient(hunting): authenticate will time out at 1354.593347s 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.278+0000 7faeda281640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7faed40a8b20 0x7faed4145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.278+0000 7faeda281640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7faed40a8b20 0x7faed4145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:56808/0 (socket says 172.21.15.7:56808) 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.278+0000 7faeda281640 1 -- 172.21.15.7:0/883701113 learned_addr learned my addr 172.21.15.7:0/883701113 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.278+0000 7faeda281640 10 monclient(hunting): get_auth_request con 0x7faed40a8b20 auth_method 0 2024-01-21T03:20:08.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.278+0000 7faeda281640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:08.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.278+0000 7faeda281640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.278+0000 7faeda281640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:08.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.278+0000 7faeda281640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:08.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.278+0000 7faeda281640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:08.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.278+0000 7faeda281640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.280+0000 7faeda281640 10 monclient(hunting): handle_auth_done global_id 4384 payload 274 2024-01-21T03:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.280+0000 7faeda281640 1 -- 172.21.15.7:0/883701113 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faed40a9d10 msgr2=0x7faed413f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.280+0000 7faeda281640 1 --2- 172.21.15.7:0/883701113 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faed40a9d10 0x7faed413f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.280+0000 7faeda281640 1 -- 172.21.15.7:0/883701113 >> v1:172.21.15.7:6789/0 conn(0x7faed40b2310 legacy=0x7faed413f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.280+0000 7faeda281640 10 monclient: _finish_hunting 0 2024-01-21T03:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.280+0000 7faeda281640 1 monclient: found mon.noname-c 2024-01-21T03:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.280+0000 7faeda281640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.280+0000 7faeda281640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.280+0000 7faeda281640 1 -- 172.21.15.7:0/883701113 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed4146230 con 0x7faed40a8b20 2024-01-21T03:20:08.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.283+0000 7faeda281640 1 --2- 172.21.15.7:0/883701113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40a8b20 0x7faed4145b80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7faed0009870 tx=0x7faed002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7343ba3279dd6893 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.287+0000 7faed927f640 1 -- 172.21.15.7:0/883701113 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed00317c0 con 0x7faed40a8b20 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.287+0000 7faed927f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.287+0000 7faed927f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.287+0000 7faed927f640 10 monclient: dump: 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.287+0000 7faed927f640 10 monclient: _finish_auth 0 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.287+0000 7faed927f640 10 monclient: _check_auth_tickets 2024-01-21T03:20:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.287+0000 7faed927f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faedc4e5640 5 monclient: authenticate success, global_id 4384 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faed927f640 1 -- 172.21.15.7:0/883701113 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed0031920 con 0x7faed40a8b20 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faedc4e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faed927f640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faed927f640 1 -- 172.21.15.7:0/883701113 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0031cb0 con 0x7faed40a8b20 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faedc4e5640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faed927f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faedc4e5640 4 set_mon_vals no callback set 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faed927f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faed927f640 10 monclient: dump: 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.276 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.276 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.276 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.276 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faedc4e5640 10 monclient: shutdown 2024-01-21T03:20:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faedc4e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faedc4e5640 1 -- 172.21.15.7:0/883701113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40a8b20 msgr2=0x7faed4145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.288+0000 7faedc4e5640 1 --2- 172.21.15.7:0/883701113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40a8b20 0x7faed4145b80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7faed0009870 tx=0x7faed002b860 comp rx=0 tx=0).stop 2024-01-21T03:20:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.289+0000 7faedc4e5640 1 -- 172.21.15.7:0/883701113 shutdown_connections 2024-01-21T03:20:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.289+0000 7faedc4e5640 1 --2- 172.21.15.7:0/883701113 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faed40a9d10 0x7faed413f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.289+0000 7faedc4e5640 1 --2- 172.21.15.7:0/883701113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40a8b20 0x7faed4145b80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.289+0000 7faedc4e5640 1 -- 172.21.15.7:0/883701113 >> 172.21.15.7:0/883701113 conn(0x7faed4066fb0 msgr2=0x7faed40b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.289+0000 7faedc4e5640 1 -- 172.21.15.7:0/883701113 shutdown_connections 2024-01-21T03:20:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.289+0000 7faedc4e5640 1 -- 172.21.15.7:0/883701113 wait complete. 2024-01-21T03:20:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.289+0000 7fcfe6d05640 10 monclient: get_monmap_and_config 2024-01-21T03:20:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.289+0000 7fcfe6d05640 10 monclient: build_initial_monmap 2024-01-21T03:20:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.290+0000 7fcfe6d05640 10 monclient: monmap: 2024-01-21T03:20:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.279 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:08.290967+0000 2024-01-21T03:20:08.279 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:08.290967+0000 2024-01-21T03:20:08.279 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:08.279 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.279 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:08.279 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:08.279 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:08.279 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:08.279 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:08.279 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.290+0000 7faedc4e5640 10 monclient: build_initial_monmap 2024-01-21T03:20:08.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.290+0000 7faedc4e5640 10 monclient: monmap: 2024-01-21T03:20:08.280 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:08.280 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:08.280 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:08.280 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:08.280 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:08.280 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.280 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:08.280 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:08.280 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:08.280 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.290+0000 7fcfe6d05640 1 Processor -- start 2024-01-21T03:20:08.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.290+0000 7faedc4e5640 1 librados: starting msgr at 2024-01-21T03:20:08.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.290+0000 7faedc4e5640 1 librados: starting objecter 2024-01-21T03:20:08.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.290+0000 7fcfe6d05640 1 -- start start 2024-01-21T03:20:08.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.290+0000 7fcfe6d05640 10 monclient: init 2024-01-21T03:20:08.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.290+0000 7faedc4e5640 1 Processor -- start 2024-01-21T03:20:08.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 1 -- start start 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 1 librados: setting wanted keys 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 1 librados: calling monclient init 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 10 monclient: init 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40a8b20 0x7faed41daf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 10 monclient(hunting): picked mon.noname-b con 0x7faed40a8b20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faed40a9d10 0x7faed4139430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 10 monclient(hunting): picked mon.noname-c con 0x7faed40a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed40b2310 0x7faed4139970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 10 monclient(hunting): picked mon.noname-a con 0x7faed40b2310 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed4139eb0 con 0x7faed40b2310 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed41db670 con 0x7faed40a8b20 2024-01-21T03:20:08.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed4141e80 con 0x7faed40a9d10 2024-01-21T03:20:08.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:08.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7faedc4e5640 10 monclient(hunting): authenticate will time out at 1354.607386s 2024-01-21T03:20:08.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:08.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-01-21T03:20:08.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcfe00a8b20 0x7fcfe0145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 10 monclient(hunting): picked mon.noname-b con 0x7fcfe00a8b20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:08.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcfe00a9d10 0x7fcfe013f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 10 monclient(hunting): picked mon.noname-c con 0x7fcfe00a9d10 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:08.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcfe00b2310 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:08.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe01456f0 con 0x7fcfe00b2310 2024-01-21T03:20:08.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe00a98c0 con 0x7fcfe00a9d10 2024-01-21T03:20:08.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe01460c0 con 0x7fcfe00a8b20 2024-01-21T03:20:08.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:08.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.291+0000 7fcfe6d05640 10 monclient(hunting): authenticate will time out at 1354.607578s 2024-01-21T03:20:08.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.292+0000 7faeda281640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40a8b20 0x7faed41daf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.292+0000 7faedaa82640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed40b2310 0x7faed4139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.292+0000 7faeda281640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40a8b20 0x7faed41daf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:56818/0 (socket says 172.21.15.7:56818) 2024-01-21T03:20:08.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.292+0000 7faeda281640 1 -- 172.21.15.7:0/1934487476 learned_addr learned my addr 172.21.15.7:0/1934487476 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:08.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.292+0000 7faeda281640 10 monclient(hunting): get_auth_request con 0x7faed40a8b20 auth_method 0 2024-01-21T03:20:08.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.292+0000 7faeda281640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:08.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.292+0000 7faeda281640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.292+0000 7faeda281640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:08.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.292+0000 7faedaa82640 10 monclient(hunting): get_auth_request con 0x7faed40b2310 auth_method 0 2024-01-21T03:20:08.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.292+0000 7faedaa82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:08.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.292+0000 7faedaa82640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.292+0000 7faedaa82640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:08.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.292+0000 7fcfdffff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcfe00a9d10 0x7fcfe013f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.293+0000 7faedaa82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:08.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.293+0000 7faedaa82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:08.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.293+0000 7faedaa82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:08.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.293+0000 7faeda281640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:08.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.293+0000 7fcfe52a2640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fcfe00b2310 0x7fcfe013f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:36912/0 (socket says 172.21.15.7:36912) 2024-01-21T03:20:08.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.293+0000 7fcfe52a2640 1 -- 172.21.15.7:0/1518985160 learned_addr learned my addr 172.21.15.7:0/1518985160 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:08.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.293+0000 7faeda281640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:08.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.293+0000 7faeda281640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:08.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.293+0000 7fcfdffff640 10 monclient(hunting): get_auth_request con 0x7fcfe00a9d10 auth_method 0 2024-01-21T03:20:08.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.293+0000 7fcfdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:08.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.293+0000 7fcfdffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.293+0000 7fcfdffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:08.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.296+0000 7faeda281640 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2024-01-21T03:20:08.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.296+0000 7faeda281640 1 -- 172.21.15.7:0/1934487476 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faed40a9d10 msgr2=0x7faed4139430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.296+0000 7faeda281640 1 --2- 172.21.15.7:0/1934487476 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faed40a9d10 0x7faed4139430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.296+0000 7faeda281640 1 -- 172.21.15.7:0/1934487476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed40b2310 msgr2=0x7faed4139970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.296+0000 7faeda281640 1 --2- 172.21.15.7:0/1934487476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed40b2310 0x7faed4139970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.296+0000 7faeda281640 10 monclient: _finish_hunting 0 2024-01-21T03:20:08.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.296+0000 7faeda281640 1 monclient: found mon.noname-b 2024-01-21T03:20:08.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.296+0000 7faeda281640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:08.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.296+0000 7faeda281640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.296+0000 7faeda281640 1 -- 172.21.15.7:0/1934487476 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed00094d0 con 0x7faed40a8b20 2024-01-21T03:20:08.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.296+0000 7faedaa82640 1 --2- 172.21.15.7:0/1934487476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed40b2310 0x7faed4139970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T03:20:08.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.297+0000 7fcfdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:08.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.297+0000 7fcfdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:08.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.297+0000 7fcfdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:08.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.299+0000 7fcfdffff640 10 monclient(hunting): handle_auth_done global_id 4390 payload 274 2024-01-21T03:20:08.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.299+0000 7fcfdffff640 1 -- 172.21.15.7:0/1518985160 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcfe00a8b20 msgr2=0x7fcfe0145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:08.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.299+0000 7fcfdffff640 1 --2- 172.21.15.7:0/1518985160 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcfe00a8b20 0x7fcfe0145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.299+0000 7fcfdffff640 1 -- 172.21.15.7:0/1518985160 >> v1:172.21.15.177:6789/0 conn(0x7fcfe00b2310 legacy=0x7fcfe013f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.299+0000 7fcfdffff640 10 monclient: _finish_hunting 0 2024-01-21T03:20:08.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.299+0000 7fcfdffff640 1 monclient: found mon.noname-c 2024-01-21T03:20:08.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.299+0000 7fcfdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:08.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.299+0000 7fcfdffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.299+0000 7fcfdffff640 1 -- 172.21.15.7:0/1518985160 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe0146230 con 0x7fcfe00a9d10 2024-01-21T03:20:08.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.301+0000 7faeda281640 1 --2- 172.21.15.7:0/1934487476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40a8b20 0x7faed41daf50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faed0000c00 tx=0x7faed00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.301+0000 7faecaffd640 10 client.?.objecter ms_handle_connect 0x7faed40a8b20 2024-01-21T03:20:08.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.301+0000 7faecaffd640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:08.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.307+0000 7fcfdffff640 1 --2- 172.21.15.7:0/1518985160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfe00a9d10 0x7fcfe013f080 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0097a0 tx=0x7fcfcc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed6ea47040249e0e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faecaffd640 1 -- 172.21.15.7:0/1934487476 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed00328e0 con 0x7faed40a8b20 2024-01-21T03:20:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faecaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faecaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faecaffd640 10 monclient: dump: 2024-01-21T03:20:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faecaffd640 10 monclient: _finish_auth 0 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faecaffd640 10 monclient: _check_auth_tickets 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faecaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faecaffd640 1 -- 172.21.15.7:0/1934487476 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed0032a60 con 0x7faed40a8b20 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faedc4e5640 5 monclient: authenticate success, global_id 4387 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faedc4e5640 10 monclient: _renew_subs 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faedc4e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faedc4e5640 1 -- 172.21.15.7:0/1934487476 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed41420f0 con 0x7faed40a8b20 2024-01-21T03:20:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faecaffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:08.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faecaffd640 1 -- 172.21.15.7:0/1934487476 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0032e10 con 0x7faed40a8b20 2024-01-21T03:20:08.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faecaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faecaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:08.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faecaffd640 10 monclient: dump: 2024-01-21T03:20:08.298 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.298 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faed8a7e640 4 set_mon_vals no callback set 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faedc4e5640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faedc4e5640 10 monclient: _renew_subs 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faedc4e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faedc4e5640 1 -- 172.21.15.7:0/1934487476 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed413bc50 con 0x7faed40a8b20 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.310+0000 7faedc4e5640 1 librados: init done 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.311+0000 7faedc4e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.311+0000 7faedc4e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.311+0000 7faedc4e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.311+0000 7faedc4e5640 1 -- 172.21.15.7:0/1934487476 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed413bea0 con 0x7faed40a8b20 2024-01-21T03:20:08.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfdf7fe640 1 -- 172.21.15.7:0/1518985160 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc02e070 con 0x7fcfe00a9d10 2024-01-21T03:20:08.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfdf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:08.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfdf7fe640 10 monclient: dump: 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfdf7fe640 10 monclient: _finish_auth 0 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfdf7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfdf7fe640 1 -- 172.21.15.7:0/1518985160 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfcc0315a0 con 0x7fcfe00a9d10 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfdf7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfdf7fe640 1 -- 172.21.15.7:0/1518985160 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc031970 con 0x7fcfe00a9d10 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfe6d05640 5 monclient: authenticate success, global_id 4390 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfe6d05640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfe6d05640 4 set_mon_vals no callback set 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfdf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.312+0000 7fcfdf7fe640 10 monclient: dump: 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.302 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0ca9f640 1 -- 172.21.15.7:0/1336940169 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf80317c0 con 0x7f1d080a9d10 2024-01-21T03:20:08.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0ca9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0ca9f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0ca9f640 10 monclient: dump: 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0ca9f640 10 monclient: _finish_auth 0 2024-01-21T03:20:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0ca9f640 10 monclient: _check_auth_tickets 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0ca9f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0ca9f640 1 -- 172.21.15.7:0/1336940169 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf8031920 con 0x7f1d080a9d10 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0ca9f640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0ca9f640 1 -- 172.21.15.7:0/1336940169 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8031cb0 con 0x7f1d080a9d10 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0ca9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0ca9f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0ca9f640 10 monclient: dump: 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0fd05640 5 monclient: authenticate success, global_id 4322 2024-01-21T03:20:08.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0fd05640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:08.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0fd05640 4 set_mon_vals no callback set 2024-01-21T03:20:08.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7fcfe6d05640 10 monclient: shutdown 2024-01-21T03:20:08.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7fcfe6d05640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:08.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7fcfe6d05640 1 -- 172.21.15.7:0/1518985160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfe00a9d10 msgr2=0x7fcfe013f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7fcfe6d05640 1 --2- 172.21.15.7:0/1518985160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfe00a9d10 0x7fcfe013f080 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0097a0 tx=0x7fcfcc02b840 comp rx=0 tx=0).stop 2024-01-21T03:20:08.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7fcfe6d05640 1 -- 172.21.15.7:0/1518985160 shutdown_connections 2024-01-21T03:20:08.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7fcfe6d05640 1 --2- 172.21.15.7:0/1518985160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfe00a9d10 0x7fcfe013f080 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7fcfe6d05640 1 --2- 172.21.15.7:0/1518985160 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcfe00a8b20 0x7fcfe0145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7fcfe6d05640 1 -- 172.21.15.7:0/1518985160 >> 172.21.15.7:0/1518985160 conn(0x7fcfe0066fb0 msgr2=0x7fcfe00b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:08.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0fd05640 10 monclient: shutdown 2024-01-21T03:20:08.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0fd05640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:08.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0fd05640 1 -- 172.21.15.7:0/1336940169 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1d080a9d10 msgr2=0x7f1d0813f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0fd05640 1 --2- 172.21.15.7:0/1336940169 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1d080a9d10 0x7f1d0813f080 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80098a0 tx=0x7f1cf802b860 comp rx=0 tx=0).stop 2024-01-21T03:20:08.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7fcfe6d05640 1 -- 172.21.15.7:0/1518985160 shutdown_connections 2024-01-21T03:20:08.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7fcfe6d05640 1 -- 172.21.15.7:0/1518985160 wait complete. 2024-01-21T03:20:08.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0fd05640 1 -- 172.21.15.7:0/1336940169 shutdown_connections 2024-01-21T03:20:08.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0fd05640 1 --2- 172.21.15.7:0/1336940169 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1d080a9d10 0x7f1d0813f080 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.313+0000 7f1d0fd05640 1 -- 172.21.15.7:0/1336940169 >> 172.21.15.7:0/1336940169 conn(0x7f1d08066fb0 msgr2=0x7f1d080b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:08.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.314+0000 7f1d0fd05640 1 -- 172.21.15.7:0/1336940169 shutdown_connections 2024-01-21T03:20:08.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.314+0000 7f1d0fd05640 1 -- 172.21.15.7:0/1336940169 wait complete. 2024-01-21T03:20:08.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.314+0000 7f1d0fd05640 10 monclient: build_initial_monmap 2024-01-21T03:20:08.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.314+0000 7f1d0fd05640 10 monclient: monmap: 2024-01-21T03:20:08.309 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:08.309 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:08.309 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:08.309 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:08.309 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:08.309 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.309 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:08.309 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:08.309 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:08.309 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.314+0000 7fcfe6d05640 10 monclient: build_initial_monmap 2024-01-21T03:20:08.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.314+0000 7fcfe6d05640 10 monclient: monmap: 2024-01-21T03:20:08.310 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:08.310 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:08.310 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:08.310 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:08.310 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:08.310 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.310 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:08.310 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:08.310 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:08.310 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.314+0000 7f1d0fd05640 1 librados: starting msgr at 2024-01-21T03:20:08.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.314+0000 7f1d0fd05640 1 librados: starting objecter 2024-01-21T03:20:08.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7f1d0fd05640 1 Processor -- start 2024-01-21T03:20:08.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7f1d0fd05640 1 -- start start 2024-01-21T03:20:08.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7f1d0fd05640 1 librados: setting wanted keys 2024-01-21T03:20:08.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7f1d0fd05640 1 librados: calling monclient init 2024-01-21T03:20:08.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7f1d0fd05640 10 monclient: init 2024-01-21T03:20:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7fcfe6d05640 1 librados: starting msgr at 2024-01-21T03:20:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7fcfe6d05640 1 librados: starting objecter 2024-01-21T03:20:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7fcfe6d05640 1 Processor -- start 2024-01-21T03:20:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7fcfe6d05640 1 -- start start 2024-01-21T03:20:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7f1d0fd05640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7f1d0fd05640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:20:08.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7fcfe6d05640 1 librados: setting wanted keys 2024-01-21T03:20:08.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7fcfe6d05640 1 librados: calling monclient init 2024-01-21T03:20:08.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.316+0000 7fcfe6d05640 10 monclient: init 2024-01-21T03:20:08.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7f1d0fd05640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d080a8b20 0x7f1d081db000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7f1d0fd05640 10 monclient(hunting): picked mon.noname-b con 0x7f1d080a8b20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:08.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7f1d0fd05640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1d080a9d10 0x7f1d08140680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7f1d0fd05640 10 monclient(hunting): picked mon.noname-c con 0x7f1d080a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:08.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.315+0000 7f1d0fd05640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d080b2310 0x7f1d08140bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.316+0000 7f1d0fd05640 10 monclient(hunting): picked mon.noname-a con 0x7f1d080b2310 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:08.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.316+0000 7f1d0fd05640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.316+0000 7f1d0fd05640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d08138850 con 0x7f1d080b2310 2024-01-21T03:20:08.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.316+0000 7f1d0fd05640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.316+0000 7f1d0fd05640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d08138a80 con 0x7f1d080a8b20 2024-01-21T03:20:08.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.316+0000 7f1d0fd05640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.316+0000 7f1d0fd05640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d08138d60 con 0x7f1d080a9d10 2024-01-21T03:20:08.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.316+0000 7f1d0fd05640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:08.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.316+0000 7f1d0fd05640 10 monclient(hunting): authenticate will time out at 1354.631810s 2024-01-21T03:20:08.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7f1d0e2a2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d080b2310 0x7f1d08140bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7fcfe6d05640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:08.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7fcfe6d05640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:20:08.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7fcfe6d05640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfe00a8b20 0x7fcfe0142c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7fcfe6d05640 10 monclient(hunting): picked mon.noname-a con 0x7fcfe00a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:08.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7fcfe6d05640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcfe00a9d10 0x7fcfe0137fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7fcfe6d05640 10 monclient(hunting): picked mon.noname-c con 0x7fcfe00a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:08.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7fcfe6d05640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfe00b2310 0x7fcfe01dad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7fcfe6d05640 10 monclient(hunting): picked mon.noname-b con 0x7fcfe00b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:08.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7fcfe6d05640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7fcfe6d05640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe0139440 con 0x7fcfe00a8b20 2024-01-21T03:20:08.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7fcfe6d05640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7f1d0e2a2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d080b2310 0x7f1d08140bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45772/0 (socket says 172.21.15.7:45772) 2024-01-21T03:20:08.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7f1d0e2a2640 1 -- 172.21.15.7:0/4049963484 learned_addr learned my addr 172.21.15.7:0/4049963484 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:08.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7f1d0e2a2640 10 monclient(hunting): get_auth_request con 0x7f1d080b2310 auth_method 0 2024-01-21T03:20:08.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7f1d0e2a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:08.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7f1d0e2a2640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.317+0000 7f1d0e2a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:08.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.318+0000 7fcfe6d05640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe01da7f0 con 0x7fcfe00b2310 2024-01-21T03:20:08.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.318+0000 7fcfe6d05640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.318+0000 7fcfe6d05640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe013fd60 con 0x7fcfe00a9d10 2024-01-21T03:20:08.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.318+0000 7fcfe6d05640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:08.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.318+0000 7fcfe6d05640 10 monclient(hunting): authenticate will time out at 1354.634029s 2024-01-21T03:20:08.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.318+0000 7fcfe4aa1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfe00a8b20 0x7fcfe0142c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.318+0000 7f1d0e2a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:08.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.318+0000 7f1d0e2a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:08.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.318+0000 7f1d0d2a0640 1 --2- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1d080a9d10 0x7f1d08140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.318+0000 7f1d0e2a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:08.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.319+0000 7fcfe4aa1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfe00a8b20 0x7fcfe0142c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45778/0 (socket says 172.21.15.7:45778) 2024-01-21T03:20:08.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.319+0000 7fcfe4aa1640 1 -- 172.21.15.7:0/2071158079 learned_addr learned my addr 172.21.15.7:0/2071158079 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:08.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.319+0000 7fcfe4aa1640 10 monclient(hunting): get_auth_request con 0x7fcfe00a8b20 auth_method 0 2024-01-21T03:20:08.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.319+0000 7fcfe4aa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:08.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.319+0000 7fcfe4aa1640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.319+0000 7fcfe4aa1640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:08.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.319+0000 7fcfdffff640 1 --2- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcfe00a9d10 0x7fcfe0137fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.321+0000 7f1d0d2a0640 10 monclient(hunting): get_auth_request con 0x7f1d080a9d10 auth_method 0 2024-01-21T03:20:08.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.321+0000 7f1d0d2a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:08.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.321+0000 7f1d0d2a0640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.321+0000 7f1d0d2a0640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:08.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.321+0000 7fcfdffff640 10 monclient(hunting): get_auth_request con 0x7fcfe00a9d10 auth_method 0 2024-01-21T03:20:08.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.321+0000 7fcfdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:08.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.321+0000 7fcfdffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.321+0000 7fcfdffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:08.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.321+0000 7fcfdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:08.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.321+0000 7fcfdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:08.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.321+0000 7fcfdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:08.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.322+0000 7f1d0e2a2640 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2024-01-21T03:20:08.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.322+0000 7f1d0e2a2640 1 -- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1d080a9d10 msgr2=0x7f1d08140680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.322+0000 7f1d0e2a2640 1 --2- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1d080a9d10 0x7f1d08140680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.322+0000 7f1d0e2a2640 1 -- 172.21.15.7:0/4049963484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d080a8b20 msgr2=0x7f1d081db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.322+0000 7f1d0e2a2640 1 --2- 172.21.15.7:0/4049963484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d080a8b20 0x7f1d081db000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.322+0000 7f1d0e2a2640 10 monclient: _finish_hunting 0 2024-01-21T03:20:08.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.322+0000 7f1d0e2a2640 1 monclient: found mon.noname-a 2024-01-21T03:20:08.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.322+0000 7f1d0e2a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:08.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.322+0000 7f1d0e2a2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:20:08.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.322+0000 7f1d0e2a2640 1 -- 172.21.15.7:0/4049963484 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf80094d0 con 0x7f1d080b2310 2024-01-21T03:20:08.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.324+0000 7fcfe4aa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:08.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.324+0000 7fcfe4aa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:08.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.324+0000 7fcfe4aa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:08.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.326+0000 7fcfdffff640 10 monclient(hunting): handle_auth_done global_id 4325 payload 824 2024-01-21T03:20:08.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.326+0000 7fcfdffff640 1 -- 172.21.15.7:0/2071158079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfe00b2310 msgr2=0x7fcfe01dad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.326+0000 7fcfdffff640 1 --2- 172.21.15.7:0/2071158079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfe00b2310 0x7fcfe01dad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.326+0000 7fcfdffff640 1 -- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfe00a8b20 msgr2=0x7fcfe0142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.326+0000 7fcfdffff640 1 --2- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfe00a8b20 0x7fcfe0142c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.326+0000 7fcfdffff640 10 monclient: _finish_hunting 0 2024-01-21T03:20:08.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.326+0000 7fcfdffff640 1 monclient: found mon.noname-c 2024-01-21T03:20:08.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.326+0000 7fcfdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:08.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.326+0000 7fcfdffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:20:08.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.326+0000 7fcfdffff640 1 -- 172.21.15.7:0/2071158079 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc009400 con 0x7fcfe00a9d10 2024-01-21T03:20:08.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.328+0000 7fcfdffff640 1 --2- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcfe00a9d10 0x7fcfe0137fb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc033000 tx=0x7fcfcc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.328+0000 7fcfdd7fa640 10 client.?.objecter ms_handle_connect 0x7fcfe00a9d10 2024-01-21T03:20:08.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.328+0000 7fcfdd7fa640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:08.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.331+0000 7f1d0e2a2640 1 --2- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d080b2310 0x7f1d08140bc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000b9e0 tx=0x7f1d0000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.331+0000 7f1cfe7fc640 10 client.?.objecter ms_handle_connect 0x7f1d080b2310 2024-01-21T03:20:08.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.331+0000 7f1cfe7fc640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:08.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-21T03:20:08.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfdd7fa640 1 -- 172.21.15.7:0/2071158079 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc032900 con 0x7fcfe00a9d10 2024-01-21T03:20:08.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfdd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfdd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:08.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfdd7fa640 10 monclient: dump: 2024-01-21T03:20:08.328 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.328 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.328 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.328 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfdd7fa640 10 monclient: _finish_auth 0 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfdd7fa640 10 monclient: _check_auth_tickets 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfdd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfe6d05640 5 monclient: authenticate success, global_id 4325 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfe6d05640 10 monclient: _renew_subs 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfe6d05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfe6d05640 1 -- 172.21.15.7:0/2071158079 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe013ff20 con 0x7fcfe00a9d10 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfe6d05640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:08.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfe6d05640 10 monclient: _renew_subs 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfe6d05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfe6d05640 1 -- 172.21.15.7:0/2071158079 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe01386d0 con 0x7fcfe00a9d10 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.332+0000 7fcfe6d05640 1 librados: init done 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.333+0000 7fcfe6d05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.333+0000 7fcfe6d05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.333+0000 7fcfe6d05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.333+0000 7fcfe6d05640 1 -- 172.21.15.7:0/2071158079 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe004e970 con 0x7fcfe00a9d10 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.335+0000 7fcfdd7fa640 1 -- 172.21.15.7:0/2071158079 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfcc032a60 con 0x7fcfe00a9d10 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.335+0000 7fcfdd7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.335+0000 7fcfdd7fa640 1 -- 172.21.15.7:0/2071158079 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc032e10 con 0x7fcfe00a9d10 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.335+0000 7fcfdd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.335+0000 7fcfdd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.335+0000 7fcfdd7fa640 10 monclient: dump: 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.330 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.331 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.331 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.331 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.331 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.331 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.331 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.331 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.331 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.335+0000 7fcfdeffd640 4 set_mon_vals no callback set 2024-01-21T03:20:08.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfe7fc640 1 -- 172.21.15.7:0/4049963484 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0000cb20 con 0x7f1d080b2310 2024-01-21T03:20:08.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:20:08.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfe7fc640 10 monclient: dump: 2024-01-21T03:20:08.332 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.332 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.332 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfe7fc640 10 monclient: _finish_auth 0 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfe7fc640 10 monclient: _check_auth_tickets 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfe7fc640 1 -- 172.21.15.7:0/4049963484 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0000cc80 con 0x7f1d080b2310 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfe7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfe7fc640 1 -- 172.21.15.7:0/4049963484 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d00014680 con 0x7f1d080b2310 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfe7fc640 10 monclient: dump: 2024-01-21T03:20:08.333 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1d0fd05640 5 monclient: authenticate success, global_id 4347 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1d0fd05640 10 monclient: _renew_subs 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1d0fd05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1d0fd05640 1 -- 172.21.15.7:0/4049963484 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d081dba70 con 0x7f1d080b2310 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1cfffff640 4 set_mon_vals no callback set 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1d0fd05640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1d0fd05640 10 monclient: _renew_subs 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1d0fd05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:08.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1d0fd05640 1 -- 172.21.15.7:0/4049963484 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d08142da0 con 0x7f1d080b2310 2024-01-21T03:20:08.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.336+0000 7f1d0fd05640 1 librados: init done 2024-01-21T03:20:08.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.337+0000 7f1d0fd05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.337+0000 7f1d0fd05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.337+0000 7f1d0fd05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:08.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.337+0000 7f1d0fd05640 1 -- 172.21.15.7:0/4049963484 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0804e970 con 0x7f1d080b2310 2024-01-21T03:20:08.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.358+0000 7f1cfe7fc640 1 -- 172.21.15.7:0/4049963484 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1d0001d3b0 con 0x7f1d080b2310 2024-01-21T03:20:08.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.359+0000 7f1cfe7fc640 1 --2- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f1cec06c8e0 0x7f1cec06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.359+0000 7f1d0daa1640 1 --2- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f1cec06c8e0 0x7f1cec06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.359+0000 7f1d0daa1640 10 monclient: get_auth_request con 0x7f1cec06c8e0 auth_method 0 2024-01-21T03:20:08.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.359+0000 7f1d0daa1640 1 --2- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f1cec06c8e0 0x7f1cec06eda0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1d04001560 tx=0x7f1d04007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.359+0000 7f1cfe7fc640 10 client.4347.objecter ms_handle_connect 0x7f1cec06c8e0 2024-01-21T03:20:08.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.360+0000 7f1cfe7fc640 1 -- 172.21.15.7:0/4049963484 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1d0000ce30 con 0x7f1d080b2310 2024-01-21T03:20:08.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.360+0000 7f1cfe7fc640 10 client.4347.objecter ms_dispatch 0x7f1d08000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:08.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.360+0000 7f1cfe7fc640 3 client.4347.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:08.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.360+0000 7f1cfe7fc640 3 client.4347.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:08.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.360+0000 7f1cfe7fc640 20 client.4347.objecter dump_active .. 0 homeless 2024-01-21T03:20:08.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-21T03:20:08.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.443+0000 7f2b7c76c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T03:20:08.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.443+0000 7f2b7c76c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T03:20:08.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.443+0000 7f2b7c76c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.443+0000 7f2b7c76c640 1 -- 172.21.15.7:0/113972815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2b7404e970 con 0x7f2b740b2310 2024-01-21T03:20:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.455+0000 7efd993b0640 10 monclient: get_monmap_and_config 2024-01-21T03:20:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.455+0000 7efd993b0640 10 monclient: build_initial_monmap 2024-01-21T03:20:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.455+0000 7efd993b0640 10 monclient: monmap: 2024-01-21T03:20:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:08.456382+0000 2024-01-21T03:20:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:08.456382+0000 2024-01-21T03:20:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:08.442 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:08.442 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.456+0000 7efd993b0640 1 Processor -- start 2024-01-21T03:20:08.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.456+0000 7efd993b0640 1 -- start start 2024-01-21T03:20:08.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.456+0000 7efd993b0640 10 monclient: init 2024-01-21T03:20:08.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:08.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-01-21T03:20:08.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd94070a10 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:08.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efd94143cf0 0x7efd9406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 10 monclient(hunting): picked mon.noname-c con 0x7efd94143cf0 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:08.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efd9406cc70 0x7efd94150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 10 monclient(hunting): picked mon.noname-b con 0x7efd9406cc70 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:08.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd940718d0 con 0x7efd94070a10 2024-01-21T03:20:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd94070df0 con 0x7efd94143cf0 2024-01-21T03:20:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9406b8e0 con 0x7efd9406cc70 2024-01-21T03:20:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.457+0000 7efd993b0640 10 monclient(hunting): authenticate will time out at 1354.773080s 2024-01-21T03:20:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.528+0000 7f2973290640 10 monclient: get_monmap_and_config 2024-01-21T03:20:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.528+0000 7f2973290640 10 monclient: build_initial_monmap 2024-01-21T03:20:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.528+0000 7f2973290640 10 monclient: monmap: 2024-01-21T03:20:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:08.529517+0000 2024-01-21T03:20:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:08.529517+0000 2024-01-21T03:20:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:08.515 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:08.515 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:08.515 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:08.515 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 1 Processor -- start 2024-01-21T03:20:08.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 1 -- start start 2024-01-21T03:20:08.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 10 monclient: init 2024-01-21T03:20:08.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:08.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-01-21T03:20:08.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f296c0a8a40 0x7f296c145b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 10 monclient(hunting): picked mon.noname-b con 0x7f296c0a8a40 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:08.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f296c0b2210 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:08.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f296c0a8e20 0x7f296c14c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 10 monclient(hunting): picked mon.noname-a con 0x7f296c0a8e20 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:08.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296c145700 con 0x7f296c0b2210 2024-01-21T03:20:08.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f296c0aa7b0 con 0x7f296c0a8e20 2024-01-21T03:20:08.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0aa9b0 con 0x7f296c0a8a40 2024-01-21T03:20:08.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:08.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.529+0000 7f2973290640 10 monclient(hunting): authenticate will time out at 1354.845670s 2024-01-21T03:20:08.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.555+0000 7faecaffd640 1 -- 172.21.15.7:0/1934487476 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7faed004d3f0 con 0x7faed40a8b20 2024-01-21T03:20:08.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.556+0000 7faecaffd640 1 --2- 172.21.15.7:0/1934487476 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7faeb806c890 0x7faeb806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.556+0000 7faed9a80640 1 --2- 172.21.15.7:0/1934487476 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7faeb806c890 0x7faeb806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.556+0000 7faed9a80640 10 monclient: get_auth_request con 0x7faeb806c890 auth_method 0 2024-01-21T03:20:08.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.556+0000 7faed9a80640 1 --2- 172.21.15.7:0/1934487476 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7faeb806c890 0x7faeb806ed50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faed413bbe0 tx=0x7faec4006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.556+0000 7faecaffd640 10 client.4387.objecter ms_handle_connect 0x7faeb806c890 2024-01-21T03:20:08.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.557+0000 7faecaffd640 1 -- 172.21.15.7:0/1934487476 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7faed007a7a0 con 0x7faed40a8b20 2024-01-21T03:20:08.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.557+0000 7faecaffd640 10 client.4387.objecter ms_dispatch 0x7faed4000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:08.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.557+0000 7faecaffd640 3 client.4387.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:08.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.557+0000 7faecaffd640 3 client.4387.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:08.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.557+0000 7faecaffd640 20 client.4387.objecter dump_active .. 0 homeless 2024-01-21T03:20:08.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.568+0000 7faecaffd640 1 -- 172.21.15.7:0/1934487476 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7faed00763e0 con 0x7faed40a8b20 2024-01-21T03:20:08.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.568+0000 7faecaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.568+0000 7faecaffd640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:08.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.569+0000 7efd927fc640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efd94143cf0 0x7efd9406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.570+0000 7efd927fc640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efd94143cf0 0x7efd9406c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:56864/0 (socket says 172.21.15.7:56864) 2024-01-21T03:20:08.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.570+0000 7efd927fc640 1 -- 172.21.15.7:0/2503415125 learned_addr learned my addr 172.21.15.7:0/2503415125 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:08.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.570+0000 7efd927fc640 10 monclient(hunting): get_auth_request con 0x7efd94143cf0 auth_method 0 2024-01-21T03:20:08.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.570+0000 7efd927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:08.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.570+0000 7efd927fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.570+0000 7efd927fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:08.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.571+0000 7efd927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:08.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.571+0000 7efd927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:08.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.571+0000 7efd927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:08.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.574+0000 7efd927fc640 10 monclient(hunting): handle_auth_done global_id 4396 payload 274 2024-01-21T03:20:08.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.574+0000 7efd927fc640 1 -- 172.21.15.7:0/2503415125 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efd9406cc70 msgr2=0x7efd94150cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:08.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.574+0000 7efd927fc640 1 --2- 172.21.15.7:0/2503415125 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efd9406cc70 0x7efd94150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.574+0000 7efd927fc640 1 -- 172.21.15.7:0/2503415125 >> v1:172.21.15.7:6789/0 conn(0x7efd94070a10 legacy=0x7efd941435e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:08.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.574+0000 7efd927fc640 10 monclient: _finish_hunting 0 2024-01-21T03:20:08.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.574+0000 7efd927fc640 1 monclient: found mon.noname-c 2024-01-21T03:20:08.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.574+0000 7efd927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:08.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.574+0000 7efd927fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.574+0000 7efd927fc640 1 -- 172.21.15.7:0/2503415125 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9406bbc0 con 0x7efd94143cf0 2024-01-21T03:20:08.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.579+0000 7f297082b640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f296c0b2210 0x7f296c13f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:36920/0 (socket says 172.21.15.7:36920) 2024-01-21T03:20:08.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.579+0000 7f297082b640 1 -- 172.21.15.7:0/2538211310 learned_addr learned my addr 172.21.15.7:0/2538211310 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:08.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.579+0000 7f2b6affd640 1 -- 172.21.15.7:0/113972815 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2b70058370 con 0x7f2b740b2310 2024-01-21T03:20:08.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.579+0000 7f2b6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T03:20:08.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.579+0000 7f2b6affd640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:08.565 INFO:teuthology.orchestra.run.smithi007.stdout:77309411337 2024-01-21T03:20:08.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 10 librados: watch_flush enter 2024-01-21T03:20:08.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 10 librados: watch_flush exit 2024-01-21T03:20:08.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 20 client.4381.objecter shutdown clearing up homeless session... 2024-01-21T03:20:08.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 10 client.4381.objecter successfully canceled tick 2024-01-21T03:20:08.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 1 -- 172.21.15.7:0/113972815 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f2b5806c8b0 msgr2=0x7f2b5806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 1 --2- 172.21.15.7:0/113972815 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f2b5806c8b0 0x7f2b5806ed70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c036000 tx=0x7f2b6c008030 comp rx=0 tx=0).stop 2024-01-21T03:20:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 10 monclient: shutdown 2024-01-21T03:20:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 1 -- 172.21.15.7:0/113972815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b740b2310 msgr2=0x7f2b740af080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 1 --2- 172.21.15.7:0/113972815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b740b2310 0x7f2b740af080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000c7a0 tx=0x7f2b7000cc60 comp rx=0 tx=0).stop 2024-01-21T03:20:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 1 -- 172.21.15.7:0/113972815 shutdown_connections 2024-01-21T03:20:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 1 --2- 172.21.15.7:0/113972815 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f2b5806c8b0 0x7f2b5806ed70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 1 --2- 172.21.15.7:0/113972815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b740b2310 0x7f2b740af080 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 1 --2- 172.21.15.7:0/113972815 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2b740a8e90 0x7f2b740aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 1 --2- 172.21.15.7:0/113972815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b740a8ab0 0x7f2b740b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 1 -- 172.21.15.7:0/113972815 >> 172.21.15.7:0/113972815 conn(0x7f2b74066f60 msgr2=0x7f2b7413ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 1 -- 172.21.15.7:0/113972815 shutdown_connections 2024-01-21T03:20:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 1 -- 172.21.15.7:0/113972815 wait complete. 2024-01-21T03:20:08.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.581+0000 7f2b7c76c640 1 librados: shutdown 2024-01-21T03:20:08.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.585+0000 7efd927fc640 1 --2- 172.21.15.7:0/2503415125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd94143cf0 0x7efd9406c730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efd88009870 tx=0x7efd8802b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68f594c5021f83da server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.587+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757814519 0 0) 0x7f296c145700 con 0x7f296c0b2210 2024-01-21T03:20:08.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.587+0000 7f2963fff640 10 monclient(hunting): _negotiate 2024-01-21T03:20:08.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.587+0000 7f2963fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.587+0000 7f2963fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:08.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.587+0000 7f2963fff640 10 monclient(hunting): my global_id is 4399 2024-01-21T03:20:08.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.587+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2958003520 con 0x7f296c0b2210 2024-01-21T03:20:08.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd91ffb640 1 -- 172.21.15.7:0/2503415125 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88031740 con 0x7efd94143cf0 2024-01-21T03:20:08.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd91ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:08.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd91ffb640 10 monclient: dump: 2024-01-21T03:20:08.577 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.577 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd91ffb640 10 monclient: _finish_auth 0 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd91ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd91ffb640 1 -- 172.21.15.7:0/2503415125 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd880318a0 con 0x7efd94143cf0 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd993b0640 5 monclient: authenticate success, global_id 4396 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd993b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:08.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd91ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd91ffb640 1 -- 172.21.15.7:0/2503415125 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88031c30 con 0x7efd94143cf0 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd993b0640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd993b0640 4 set_mon_vals no callback set 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd91ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.591+0000 7efd91ffb640 10 monclient: dump: 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.579 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.580 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.581 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411337 got 77309411337 for osd.6 2024-01-21T03:20:08.581 DEBUG:teuthology.parallel:result is None 2024-01-21T03:20:08.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7efd993b0640 10 monclient: shutdown 2024-01-21T03:20:08.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7efd993b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:08.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7efd993b0640 1 -- 172.21.15.7:0/2503415125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd94143cf0 msgr2=0x7efd9406c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7efd993b0640 1 --2- 172.21.15.7:0/2503415125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd94143cf0 0x7efd9406c730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efd88009870 tx=0x7efd8802b7e0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7efd993b0640 1 -- 172.21.15.7:0/2503415125 shutdown_connections 2024-01-21T03:20:08.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7efd993b0640 1 --2- 172.21.15.7:0/2503415125 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efd9406cc70 0x7efd94150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7efd993b0640 1 --2- 172.21.15.7:0/2503415125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd94143cf0 0x7efd9406c730 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7efd993b0640 1 -- 172.21.15.7:0/2503415125 >> 172.21.15.7:0/2503415125 conn(0x7efd94066f50 msgr2=0x7efd940b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:08.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 353287196 0 0) 0x7f2958003520 con 0x7f296c0b2210 2024-01-21T03:20:08.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f296c0a8a40 msgr2=0x7f296c145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:08.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7efd993b0640 1 -- 172.21.15.7:0/2503415125 shutdown_connections 2024-01-21T03:20:08.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 --2- 172.21.15.7:0/2538211310 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f296c0a8a40 0x7f296c145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 >> v2:172.21.15.7:3300/0 conn(0x7f296c0a8e20 msgr2=0x7f296c14c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:08.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 --2- 172.21.15.7:0/2538211310 >> v2:172.21.15.7:3300/0 conn(0x7f296c0a8e20 0x7f296c14c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient: _finish_hunting 0 2024-01-21T03:20:08.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 monclient: found mon.noname-c-legacy 2024-01-21T03:20:08.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:08.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.177:6789/0 2024-01-21T03:20:08.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c0aac90 con 0x7f296c0b2210 2024-01-21T03:20:08.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient: _finish_auth 0 2024-01-21T03:20:08.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient: _check_auth_tickets 2024-01-21T03:20:08.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:08.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 749591385 0 0) 0x7f295c001670 con 0x7f296c0b2210 2024-01-21T03:20:08.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-01-21T03:20:08.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7efd993b0640 1 -- 172.21.15.7:0/2503415125 wait complete. 2024-01-21T03:20:08.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient: dump: 2024-01-21T03:20:08.585 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] but i'm connected to v1:172.21.15.177:6789/0, reconnecting 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 >> v1:172.21.15.177:6789/0 conn(0x7f296c0b2210 legacy=0x7f296c13f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 --2- 172.21.15.7:0/2538211310 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f296c0a8a40 0x7f2958003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient(hunting): picked mon.c con 0x7f296c0a8a40 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 --2- 172.21.15.7:0/2538211310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29580040d0 0x7f29580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient(hunting): picked mon.a con 0x7f29580040d0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:08.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 --2- 172.21.15.7:0/2538211310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2958004e20 0x7f295800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient(hunting): picked mon.b con 0x7f2958004e20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:08.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0aa7b0 con 0x7f29580040d0 2024-01-21T03:20:08.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0aa9b0 con 0x7f2958004e20 2024-01-21T03:20:08.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29580055a0 con 0x7f296c0a8a40 2024-01-21T03:20:08.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:08.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.592+0000 7f2963fff640 10 monclient(hunting): _finish_auth 0 2024-01-21T03:20:08.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.593+0000 7efd993b0640 10 monclient: build_initial_monmap 2024-01-21T03:20:08.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.593+0000 7efd993b0640 10 monclient: monmap: 2024-01-21T03:20:08.588 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:08.588 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:08.588 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:08.588 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:08.588 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:08.588 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.589 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:08.589 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:08.589 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:08.589 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.594+0000 7efd993b0640 1 librados: starting msgr at 2024-01-21T03:20:08.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.594+0000 7efd993b0640 1 librados: starting objecter 2024-01-21T03:20:08.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.594+0000 7efd993b0640 1 Processor -- start 2024-01-21T03:20:08.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.594+0000 7efd993b0640 1 -- start start 2024-01-21T03:20:08.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.594+0000 7efd993b0640 1 librados: setting wanted keys 2024-01-21T03:20:08.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.594+0000 7efd993b0640 1 librados: calling monclient init 2024-01-21T03:20:08.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.594+0000 7efd993b0640 10 monclient: init 2024-01-21T03:20:08.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.594+0000 7f297182d640 1 --2- 172.21.15.7:0/2538211310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2958004e20 0x7f295800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efd9406cc70 0x7efd940aa180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 10 monclient(hunting): picked mon.noname-c con 0x7efd9406cc70 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd94070a10 0x7efd94146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 10 monclient(hunting): picked mon.noname-b con 0x7efd94070a10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd94143cf0 0x7efd941df010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 10 monclient(hunting): picked mon.noname-a con 0x7efd94143cf0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd940ab830 con 0x7efd94143cf0 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd940ac850 con 0x7efd94070a10 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd940ab310 con 0x7efd9406cc70 2024-01-21T03:20:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.595+0000 7efd993b0640 10 monclient(hunting): authenticate will time out at 1354.911512s 2024-01-21T03:20:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.596+0000 7f297182d640 10 monclient(hunting): get_auth_request con 0x7f2958004e20 auth_method 0 2024-01-21T03:20:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.596+0000 7f297182d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.596+0000 7f297182d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.596+0000 7f297182d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:20:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.597+0000 7f297182d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.597+0000 7f297182d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.597+0000 7f297182d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T03:20:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f297182d640 10 monclient(hunting): handle_auth_done global_id 4399 payload 293 2024-01-21T03:20:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f297182d640 1 -- 172.21.15.7:0/2538211310 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f296c0a8a40 msgr2=0x7f2958003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f297182d640 1 --2- 172.21.15.7:0/2538211310 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f296c0a8a40 0x7f2958003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f297182d640 1 -- 172.21.15.7:0/2538211310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29580040d0 msgr2=0x7f29580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f297182d640 1 --2- 172.21.15.7:0/2538211310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29580040d0 0x7f29580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f297182d640 10 monclient: _finish_hunting 0 2024-01-21T03:20:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f297182d640 1 monclient: found mon.b 2024-01-21T03:20:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f297182d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f297182d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f297182d640 1 -- 172.21.15.7:0/2538211310 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2958003520 con 0x7f2958004e20 2024-01-21T03:20:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f297182d640 10 monclient: _finish_auth 0 2024-01-21T03:20:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f297182d640 10 monclient: _check_auth_tickets 2024-01-21T03:20:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f297182d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f2973290640 5 monclient: authenticate success, global_id 4399 2024-01-21T03:20:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.599+0000 7f2973290640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.601+0000 7f297182d640 1 --2- 172.21.15.7:0/2538211310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2958004e20 0x7f295800b980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2964009e10 tx=0x7f29580057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65d4b0114597978 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.603+0000 7efd927fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd94070a10 0x7efd94146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.604+0000 7efd927fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd94070a10 0x7efd94146930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:56886/0 (socket says 172.21.15.7:56886) 2024-01-21T03:20:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.604+0000 7efd927fc640 1 -- 172.21.15.7:0/3420912132 learned_addr learned my addr 172.21.15.7:0/3420912132 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.604+0000 7efd927fc640 10 monclient(hunting): get_auth_request con 0x7efd94070a10 auth_method 0 2024-01-21T03:20:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.604+0000 7efd927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.604+0000 7efd927fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.604+0000 7efd927fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7efd927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7efd927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7efd927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296402cd50 con 0x7f2958004e20 2024-01-21T03:20:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2963fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2963fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2963fff640 10 monclient: dump: 2024-01-21T03:20:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2973290640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2964034440 con 0x7f2958004e20 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2963fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2963fff640 1 -- 172.21.15.7:0/2538211310 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29640347f0 con 0x7f2958004e20 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2963fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2963fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2963fff640 10 monclient: dump: 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2973290640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2973290640 4 set_mon_vals no callback set 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2973290640 10 monclient: shutdown 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2973290640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.605+0000 7f2973290640 1 -- 172.21.15.7:0/2538211310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2958004e20 msgr2=0x7f295800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.606+0000 7f2973290640 1 --2- 172.21.15.7:0/2538211310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2958004e20 0x7f295800b980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2964009e10 tx=0x7f29580057f0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.606+0000 7f297182d640 1 -- 172.21.15.7:0/2538211310 reap_dead start 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.606+0000 7f2973290640 1 -- 172.21.15.7:0/2538211310 shutdown_connections 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.606+0000 7f2973290640 1 -- 172.21.15.7:0/2538211310 >> 172.21.15.7:0/2538211310 conn(0x7f296c066f10 msgr2=0x7f296c0b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.606+0000 7f2973290640 1 -- 172.21.15.7:0/2538211310 shutdown_connections 2024-01-21T03:20:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.606+0000 7f2973290640 1 -- 172.21.15.7:0/2538211310 wait complete. 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.606+0000 7f2973290640 10 monclient: build_initial_monmap 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.606+0000 7f2973290640 10 monclient: monmap: 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 1 librados: starting msgr at 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 1 librados: starting objecter 2024-01-21T03:20:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 1 Processor -- start 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 1 -- start start 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 1 librados: setting wanted keys 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 1 librados: calling monclient init 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 10 monclient: init 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2958004e20 0x7f296c13fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 10 monclient(hunting): picked mon.noname-a con 0x7f2958004e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f296c144350 0x7f296c1dae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 10 monclient(hunting): picked mon.noname-b con 0x7f296c144350 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f296c1417d0 0x7f296c1db350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 10 monclient(hunting): picked mon.noname-c con 0x7f296c1417d0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c141490 con 0x7f2958004e20 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c145310 con 0x7f296c144350 2024-01-21T03:20:08.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:08.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296c144df0 con 0x7f296c1417d0 2024-01-21T03:20:08.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:08.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.607+0000 7f2973290640 10 monclient(hunting): authenticate will time out at 1354.923691s 2024-01-21T03:20:08.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.608+0000 7efd927fc640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2024-01-21T03:20:08.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.608+0000 7efd927fc640 1 -- 172.21.15.7:0/3420912132 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efd9406cc70 msgr2=0x7efd940aa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.608+0000 7efd927fc640 1 --2- 172.21.15.7:0/3420912132 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efd9406cc70 0x7efd940aa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.608+0000 7efd927fc640 1 -- 172.21.15.7:0/3420912132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd94143cf0 msgr2=0x7efd941df010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.608+0000 7efd927fc640 1 --2- 172.21.15.7:0/3420912132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd94143cf0 0x7efd941df010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.608+0000 7efd927fc640 10 monclient: _finish_hunting 0 2024-01-21T03:20:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.608+0000 7efd927fc640 1 monclient: found mon.noname-b 2024-01-21T03:20:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.608+0000 7efd927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.608+0000 7efd927fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.608+0000 7efd927fc640 1 -- 172.21.15.7:0/3420912132 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd880094d0 con 0x7efd94070a10 2024-01-21T03:20:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.610+0000 7efd927fc640 1 --2- 172.21.15.7:0/3420912132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd94070a10 0x7efd94146930 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efd880090b0 tx=0x7efd880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.610+0000 7efd7b7fe640 10 client.?.objecter ms_handle_connect 0x7efd94070a10 2024-01-21T03:20:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.610+0000 7efd7b7fe640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:08.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.611+0000 7f297082b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f296c144350 0x7f296c1dae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.611+0000 7fcfdd7fa640 1 -- 172.21.15.7:0/2071158079 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fcfcc04e470 con 0x7fcfe00a9d10 2024-01-21T03:20:08.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.612+0000 7fcfdd7fa640 1 --2- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fcfc406c860 0x7fcfc406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.612+0000 7fcfe4aa1640 1 --2- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fcfc406c860 0x7fcfc406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.612+0000 7fcfe4aa1640 10 monclient: get_auth_request con 0x7fcfc406c860 auth_method 0 2024-01-21T03:20:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.612+0000 7f297082b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f296c144350 0x7f296c1dae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:56898/0 (socket says 172.21.15.7:56898) 2024-01-21T03:20:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.612+0000 7f297082b640 1 -- 172.21.15.7:0/3447816120 learned_addr learned my addr 172.21.15.7:0/3447816120 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.612+0000 7f297082b640 10 monclient(hunting): get_auth_request con 0x7f296c144350 auth_method 0 2024-01-21T03:20:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.612+0000 7f297082b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.612+0000 7f297082b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.612+0000 7f297082b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.612+0000 7fcfdd7fa640 10 client.4325.objecter ms_handle_connect 0x7fcfc406c860 2024-01-21T03:20:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.612+0000 7fcfe4aa1640 1 --2- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fcfc406c860 0x7fcfc406ed20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009a40 tx=0x7fcfd40091c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.613+0000 7f297082b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.613+0000 7f297082b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.613+0000 7f297082b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.617+0000 7f297082b640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2024-01-21T03:20:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.617+0000 7f297082b640 1 -- 172.21.15.7:0/3447816120 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f296c1417d0 msgr2=0x7f296c1db350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.617+0000 7f297082b640 1 --2- 172.21.15.7:0/3447816120 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f296c1417d0 0x7f296c1db350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.617+0000 7f297082b640 1 -- 172.21.15.7:0/3447816120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2958004e20 msgr2=0x7f296c13fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.617+0000 7f297082b640 1 --2- 172.21.15.7:0/3447816120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2958004e20 0x7f296c13fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.617+0000 7f297082b640 10 monclient: _finish_hunting 0 2024-01-21T03:20:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.617+0000 7f297082b640 1 monclient: found mon.noname-b 2024-01-21T03:20:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.617+0000 7f297082b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.617+0000 7f297082b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.617+0000 7f297082b640 1 -- 172.21.15.7:0/3447816120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c1db890 con 0x7f296c144350 2024-01-21T03:20:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd7b7fe640 1 -- 172.21.15.7:0/3420912132 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88033c70 con 0x7efd94070a10 2024-01-21T03:20:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd7b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd7b7fe640 10 monclient: dump: 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd7b7fe640 10 monclient: _finish_auth 0 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd7b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd7b7fe640 1 -- 172.21.15.7:0/3420912132 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd88033df0 con 0x7efd94070a10 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd7b7fe640 1 -- 172.21.15.7:0/3420912132 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8803d6b0 con 0x7efd94070a10 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd993b0640 5 monclient: authenticate success, global_id 4402 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd7b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd7b7fe640 10 monclient: dump: 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd993b0640 10 monclient: _renew_subs 2024-01-21T03:20:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd917fa640 4 set_mon_vals no callback set 2024-01-21T03:20:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd993b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd993b0640 1 -- 172.21.15.7:0/3420912132 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd940aadc0 con 0x7efd94070a10 2024-01-21T03:20:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd993b0640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd993b0640 10 monclient: _renew_subs 2024-01-21T03:20:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd993b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd993b0640 1 -- 172.21.15.7:0/3420912132 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd941df550 con 0x7efd94070a10 2024-01-21T03:20:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.619+0000 7efd993b0640 1 librados: init done 2024-01-21T03:20:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.620+0000 7efd993b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.620+0000 7efd993b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.620+0000 7efd993b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.620+0000 7efd993b0640 1 -- 172.21.15.7:0/3420912132 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd940ab530 con 0x7efd94070a10 2024-01-21T03:20:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.625+0000 7fcfdd7fa640 1 -- 172.21.15.7:0/2071158079 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcfcc07a8e0 con 0x7fcfe00a9d10 2024-01-21T03:20:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.625+0000 7fcfdd7fa640 10 client.4325.objecter ms_dispatch 0x7fcfe0000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.625+0000 7fcfdd7fa640 3 client.4325.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.625+0000 7fcfdd7fa640 3 client.4325.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.625+0000 7fcfdd7fa640 20 client.4325.objecter dump_active .. 0 homeless 2024-01-21T03:20:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.625+0000 7fcfdd7fa640 1 -- 172.21.15.7:0/2071158079 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fcfcc076520 con 0x7fcfe00a9d10 2024-01-21T03:20:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.625+0000 7fcfdd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.625+0000 7fcfdd7fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:08.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.697+0000 7f1cfe7fc640 1 -- 172.21.15.7:0/4049963484 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1d000548a0 con 0x7f1d080b2310 2024-01-21T03:20:08.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.697+0000 7f1cfe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.697+0000 7f1cfe7fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:08.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.816+0000 7faedc4e5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T03:20:08.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.816+0000 7faedc4e5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T03:20:08.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.816+0000 7faedc4e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.816+0000 7faedc4e5640 1 -- 172.21.15.7:0/1934487476 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7faed413bea0 con 0x7faed40a8b20 2024-01-21T03:20:08.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.821+0000 7fcfe6d05640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T03:20:08.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.821+0000 7fcfe6d05640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T03:20:08.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.821+0000 7fcfe6d05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:08.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.821+0000 7fcfe6d05640 1 -- 172.21.15.7:0/2071158079 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fcfe01da9b0 con 0x7fcfe00a9d10 2024-01-21T03:20:08.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.826+0000 7fcfdd7fa640 1 -- 172.21.15.7:0/2071158079 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcfcc079b40 con 0x7fcfe00a9d10 2024-01-21T03:20:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.826+0000 7fcfdd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T03:20:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.826+0000 7fcfdd7fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:08.812 INFO:teuthology.orchestra.run.smithi007.stdout:90194313226 2024-01-21T03:20:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 10 librados: watch_flush enter 2024-01-21T03:20:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 10 librados: watch_flush exit 2024-01-21T03:20:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 20 client.4325.objecter shutdown clearing up homeless session... 2024-01-21T03:20:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 10 client.4325.objecter successfully canceled tick 2024-01-21T03:20:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 1 -- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fcfc406c860 msgr2=0x7fcfc406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 1 --2- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fcfc406c860 0x7fcfc406ed20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009a40 tx=0x7fcfd40091c0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 10 monclient: shutdown 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 1 -- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcfe00a9d10 msgr2=0x7fcfe0137fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 1 --2- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcfe00a9d10 0x7fcfe0137fb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc033000 tx=0x7fcfcc0046c0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 1 -- 172.21.15.7:0/2071158079 shutdown_connections 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 1 --2- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fcfc406c860 0x7fcfc406ed20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 1 --2- 172.21.15.7:0/2071158079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfe00b2310 0x7fcfe01dad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 1 --2- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcfe00a9d10 0x7fcfe0137fb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 1 --2- 172.21.15.7:0/2071158079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfe00a8b20 0x7fcfe0142c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 1 -- 172.21.15.7:0/2071158079 >> 172.21.15.7:0/2071158079 conn(0x7fcfe0066fb0 msgr2=0x7fcfe00b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 1 -- 172.21.15.7:0/2071158079 shutdown_connections 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 1 -- 172.21.15.7:0/2071158079 wait complete. 2024-01-21T03:20:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.828+0000 7fcfe6d05640 1 librados: shutdown 2024-01-21T03:20:08.825 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313226 for osd.0 2024-01-21T03:20:08.825 DEBUG:teuthology.parallel:result is None 2024-01-21T03:20:08.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.860+0000 7efd7b7fe640 1 -- 172.21.15.7:0/3420912132 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7efd8803a040 con 0x7efd94070a10 2024-01-21T03:20:08.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.860+0000 7efd7b7fe640 1 --2- 172.21.15.7:0/3420912132 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7efd8006c8b0 0x7efd8006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:08.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.860+0000 7efd92ffd640 1 --2- 172.21.15.7:0/3420912132 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7efd8006c8b0 0x7efd8006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:08.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.860+0000 7efd92ffd640 10 monclient: get_auth_request con 0x7efd8006c8b0 auth_method 0 2024-01-21T03:20:08.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.861+0000 7efd7b7fe640 10 client.4402.objecter ms_handle_connect 0x7efd8006c8b0 2024-01-21T03:20:08.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.861+0000 7efd92ffd640 1 --2- 172.21.15.7:0/3420912132 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7efd8006c8b0 0x7efd8006ed70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efd84001560 tx=0x7efd84007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.863+0000 7efd7b7fe640 1 -- 172.21.15.7:0/3420912132 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efd8807aeb0 con 0x7efd94070a10 2024-01-21T03:20:08.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.863+0000 7efd7b7fe640 10 client.4402.objecter ms_dispatch 0x7efd94000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:08.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.863+0000 7efd7b7fe640 3 client.4402.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:08.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.863+0000 7efd7b7fe640 3 client.4402.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:08.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.863+0000 7efd7b7fe640 20 client.4402.objecter dump_active .. 0 homeless 2024-01-21T03:20:08.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.873+0000 7efd7b7fe640 1 -- 172.21.15.7:0/3420912132 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7efd88076af0 con 0x7efd94070a10 2024-01-21T03:20:08.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.873+0000 7efd7b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.873+0000 7efd7b7fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:08.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.873+0000 7f297082b640 1 --2- 172.21.15.7:0/3447816120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f296c144350 0x7f296c1dae10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f295c00c770 tx=0x7f295c00cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:08.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.873+0000 7f2961ffb640 10 client.?.objecter ms_handle_connect 0x7f296c144350 2024-01-21T03:20:08.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.873+0000 7f2961ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:08.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.881+0000 7faecaffd640 1 -- 172.21.15.7:0/1934487476 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faed413bea0 con 0x7faed40a8b20 2024-01-21T03:20:08.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.881+0000 7faecaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T03:20:08.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.881+0000 7faecaffd640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:08.867 INFO:teuthology.orchestra.run.smithi007.stdout:60129542152 2024-01-21T03:20:08.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 10 librados: watch_flush enter 2024-01-21T03:20:08.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 10 librados: watch_flush exit 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 20 client.4387.objecter shutdown clearing up homeless session... 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 10 client.4387.objecter successfully canceled tick 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 1 -- 172.21.15.7:0/1934487476 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7faeb806c890 msgr2=0x7faeb806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 1 --2- 172.21.15.7:0/1934487476 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7faeb806c890 0x7faeb806ed50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faed413bbe0 tx=0x7faec4006c90 comp rx=0 tx=0).stop 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 10 monclient: shutdown 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 1 -- 172.21.15.7:0/1934487476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40a8b20 msgr2=0x7faed41daf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 1 --2- 172.21.15.7:0/1934487476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40a8b20 0x7faed41daf50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faed0000c00 tx=0x7faed00046c0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 1 -- 172.21.15.7:0/1934487476 shutdown_connections 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 1 --2- 172.21.15.7:0/1934487476 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7faeb806c890 0x7faeb806ed50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 1 --2- 172.21.15.7:0/1934487476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed40b2310 0x7faed4139970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 1 --2- 172.21.15.7:0/1934487476 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faed40a9d10 0x7faed4139430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 1 --2- 172.21.15.7:0/1934487476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40a8b20 0x7faed41daf50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 1 -- 172.21.15.7:0/1934487476 >> 172.21.15.7:0/1934487476 conn(0x7faed4066fb0 msgr2=0x7faed40b33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 1 -- 172.21.15.7:0/1934487476 shutdown_connections 2024-01-21T03:20:08.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.884+0000 7faedc4e5640 1 -- 172.21.15.7:0/1934487476 wait complete. 2024-01-21T03:20:08.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.885+0000 7faedc4e5640 1 librados: shutdown 2024-01-21T03:20:08.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2961ffb640 1 -- 172.21.15.7:0/3447816120 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c0074e0 con 0x7f296c144350 2024-01-21T03:20:08.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2961ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2961ffb640 10 monclient: dump: 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2961ffb640 10 monclient: _finish_auth 0 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2961ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2961ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:08.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2973290640 5 monclient: authenticate success, global_id 4405 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2973290640 10 monclient: _renew_subs 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2973290640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2973290640 1 -- 172.21.15.7:0/3447816120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296c1dbae0 con 0x7f296c144350 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2973290640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2961ffb640 1 -- 172.21.15.7:0/3447816120 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c007640 con 0x7f296c144350 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2961ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2961ffb640 1 -- 172.21.15.7:0/3447816120 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c0074e0 con 0x7f296c144350 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2961ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2961ffb640 10 monclient: dump: 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:08.879 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:08.880 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:08.880 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:08.880 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:08.880 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:08.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f29637fe640 4 set_mon_vals no callback set 2024-01-21T03:20:08.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2973290640 10 monclient: _renew_subs 2024-01-21T03:20:08.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2973290640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2973290640 1 -- 172.21.15.7:0/3447816120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296c1d7880 con 0x7f296c144350 2024-01-21T03:20:08.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.891+0000 7f2973290640 1 librados: init done 2024-01-21T03:20:08.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.893+0000 7f2973290640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.893+0000 7f2973290640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:08.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.894+0000 7f2973290640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:08.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.894+0000 7f2973290640 1 -- 172.21.15.7:0/3447816120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296c145030 con 0x7f296c144350 2024-01-21T03:20:08.885 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.4 2024-01-21T03:20:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.899+0000 7f1d0fd05640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T03:20:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.899+0000 7f1d0fd05640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T03:20:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.899+0000 7f1d0fd05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:20:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.899+0000 7f1d0fd05640 1 -- 172.21.15.7:0/4049963484 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1d080b5b90 con 0x7f1d080b2310 2024-01-21T03:20:08.886 DEBUG:teuthology.parallel:result is None 2024-01-21T03:20:08.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.952+0000 7f1cfe7fc640 1 -- 172.21.15.7:0/4049963484 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1d00019020 con 0x7f1d080b2310 2024-01-21T03:20:08.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.952+0000 7f1cfe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T03:20:08.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.952+0000 7f1cfe7fc640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:08.938 INFO:teuthology.orchestra.run.smithi007.stdout:94489280518 2024-01-21T03:20:08.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 10 librados: watch_flush enter 2024-01-21T03:20:08.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 10 librados: watch_flush exit 2024-01-21T03:20:08.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 20 client.4347.objecter shutdown clearing up homeless session... 2024-01-21T03:20:08.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 10 client.4347.objecter successfully canceled tick 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 1 -- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f1cec06c8e0 msgr2=0x7f1cec06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 1 --2- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f1cec06c8e0 0x7f1cec06eda0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1d04001560 tx=0x7f1d04007990 comp rx=0 tx=0).stop 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 10 monclient: shutdown 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 1 -- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d080b2310 msgr2=0x7f1d08140bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 1 --2- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d080b2310 0x7f1d08140bc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000b9e0 tx=0x7f1d0000bea0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 1 -- 172.21.15.7:0/4049963484 shutdown_connections 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 1 --2- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f1cec06c8e0 0x7f1cec06eda0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 1 --2- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d080b2310 0x7f1d08140bc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 1 --2- 172.21.15.7:0/4049963484 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1d080a9d10 0x7f1d08140680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 1 --2- 172.21.15.7:0/4049963484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d080a8b20 0x7f1d081db000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 1 -- 172.21.15.7:0/4049963484 >> 172.21.15.7:0/4049963484 conn(0x7f1d08066fb0 msgr2=0x7f1d080b3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 1 -- 172.21.15.7:0/4049963484 shutdown_connections 2024-01-21T03:20:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 1 -- 172.21.15.7:0/4049963484 wait complete. 2024-01-21T03:20:08.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:08.955+0000 7f1d0fd05640 1 librados: shutdown 2024-01-21T03:20:08.955 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280518 for osd.3 2024-01-21T03:20:08.955 DEBUG:teuthology.parallel:result is None 2024-01-21T03:20:09.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.071+0000 7efd993b0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T03:20:09.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.071+0000 7efd993b0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T03:20:09.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.071+0000 7efd993b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:09.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.071+0000 7efd993b0640 1 -- 172.21.15.7:0/3420912132 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7efd940ac990 con 0x7efd94070a10 2024-01-21T03:20:09.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.138+0000 7f2961ffb640 1 -- 172.21.15.7:0/3447816120 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f295c0047e0 con 0x7f296c144350 2024-01-21T03:20:09.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.138+0000 7f2961ffb640 1 --2- 172.21.15.7:0/3447816120 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f295006c8c0 0x7f295006ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:09.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.138+0000 7f297102c640 1 --2- 172.21.15.7:0/3447816120 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f295006c8c0 0x7f295006ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:09.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.138+0000 7f297102c640 10 monclient: get_auth_request con 0x7f295006c8c0 auth_method 0 2024-01-21T03:20:09.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.139+0000 7f2961ffb640 10 client.4405.objecter ms_handle_connect 0x7f295006c8c0 2024-01-21T03:20:09.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.139+0000 7f297102c640 1 --2- 172.21.15.7:0/3447816120 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f295006c8c0 0x7f295006ed80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f29680096b0 tx=0x7f2968006ca0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:09.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.141+0000 7f2961ffb640 1 -- 172.21.15.7:0/3447816120 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f295c057b50 con 0x7f296c144350 2024-01-21T03:20:09.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.141+0000 7f2961ffb640 10 client.4405.objecter ms_dispatch 0x7f296c000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:09.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.141+0000 7f2961ffb640 3 client.4405.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:09.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.141+0000 7f2961ffb640 3 client.4405.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:09.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.141+0000 7f2961ffb640 20 client.4405.objecter dump_active .. 0 homeless 2024-01-21T03:20:09.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.150+0000 7f2961ffb640 1 -- 172.21.15.7:0/3447816120 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f295c053790 con 0x7f296c144350 2024-01-21T03:20:09.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.150+0000 7f2961ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:09.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.150+0000 7f2961ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:09.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.200+0000 7efd7b7fe640 1 -- 172.21.15.7:0/3420912132 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efd940ac990 con 0x7efd94070a10 2024-01-21T03:20:09.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.200+0000 7efd7b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T03:20:09.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.200+0000 7efd7b7fe640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:09.187 INFO:teuthology.orchestra.run.smithi007.stdout:77309411339 2024-01-21T03:20:09.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 10 librados: watch_flush enter 2024-01-21T03:20:09.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 10 librados: watch_flush exit 2024-01-21T03:20:09.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 20 client.4402.objecter shutdown clearing up homeless session... 2024-01-21T03:20:09.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 10 client.4402.objecter successfully canceled tick 2024-01-21T03:20:09.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 1 -- 172.21.15.7:0/3420912132 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7efd8006c8b0 msgr2=0x7efd8006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:09.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 1 --2- 172.21.15.7:0/3420912132 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7efd8006c8b0 0x7efd8006ed70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efd84001560 tx=0x7efd84007990 comp rx=0 tx=0).stop 2024-01-21T03:20:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 10 monclient: shutdown 2024-01-21T03:20:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 1 -- 172.21.15.7:0/3420912132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd94070a10 msgr2=0x7efd94146930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 1 --2- 172.21.15.7:0/3420912132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd94070a10 0x7efd94146930 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efd880090b0 tx=0x7efd880046c0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 1 -- 172.21.15.7:0/3420912132 shutdown_connections 2024-01-21T03:20:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 1 --2- 172.21.15.7:0/3420912132 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7efd8006c8b0 0x7efd8006ed70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 1 --2- 172.21.15.7:0/3420912132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd94143cf0 0x7efd941df010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 1 --2- 172.21.15.7:0/3420912132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd94070a10 0x7efd94146930 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.202+0000 7efd993b0640 1 --2- 172.21.15.7:0/3420912132 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efd9406cc70 0x7efd940aa180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.203+0000 7efd993b0640 1 -- 172.21.15.7:0/3420912132 >> 172.21.15.7:0/3420912132 conn(0x7efd94066f50 msgr2=0x7efd940b54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.203+0000 7efd993b0640 1 -- 172.21.15.7:0/3420912132 shutdown_connections 2024-01-21T03:20:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.203+0000 7efd993b0640 1 -- 172.21.15.7:0/3420912132 wait complete. 2024-01-21T03:20:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.203+0000 7efd993b0640 1 librados: shutdown 2024-01-21T03:20:09.199 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411338 got 77309411339 for osd.5 2024-01-21T03:20:09.199 DEBUG:teuthology.parallel:result is None 2024-01-21T03:20:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.335+0000 7f2973290640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T03:20:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.336+0000 7f2973290640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T03:20:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.336+0000 7f2973290640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:09.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.336+0000 7f2973290640 1 -- 172.21.15.7:0/3447816120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f296c145030 con 0x7f296c144350 2024-01-21T03:20:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.339+0000 7f2961ffb640 1 -- 172.21.15.7:0/3447816120 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f296c145030 con 0x7f296c144350 2024-01-21T03:20:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.339+0000 7f2961ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T03:20:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.339+0000 7f2961ffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:09.325 INFO:teuthology.orchestra.run.smithi007.stdout:90194313225 2024-01-21T03:20:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 10 librados: watch_flush enter 2024-01-21T03:20:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 10 librados: watch_flush exit 2024-01-21T03:20:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 20 client.4405.objecter shutdown clearing up homeless session... 2024-01-21T03:20:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 10 client.4405.objecter successfully canceled tick 2024-01-21T03:20:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 1 -- 172.21.15.7:0/3447816120 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f295006c8c0 msgr2=0x7f295006ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 1 --2- 172.21.15.7:0/3447816120 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f295006c8c0 0x7f295006ed80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f29680096b0 tx=0x7f2968006ca0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 10 monclient: shutdown 2024-01-21T03:20:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 1 -- 172.21.15.7:0/3447816120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f296c144350 msgr2=0x7f296c1dae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 1 --2- 172.21.15.7:0/3447816120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f296c144350 0x7f296c1dae10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f295c00c770 tx=0x7f295c00cc30 comp rx=0 tx=0).stop 2024-01-21T03:20:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 1 -- 172.21.15.7:0/3447816120 shutdown_connections 2024-01-21T03:20:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 1 --2- 172.21.15.7:0/3447816120 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f295006c8c0 0x7f295006ed80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 1 --2- 172.21.15.7:0/3447816120 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f296c1417d0 0x7f296c1db350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 1 --2- 172.21.15.7:0/3447816120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f296c144350 0x7f296c1dae10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 1 --2- 172.21.15.7:0/3447816120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2958004e20 0x7f296c13fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 1 -- 172.21.15.7:0/3447816120 >> 172.21.15.7:0/3447816120 conn(0x7f296c066f10 msgr2=0x7f296c0b5650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 1 -- 172.21.15.7:0/3447816120 shutdown_connections 2024-01-21T03:20:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 1 -- 172.21.15.7:0/3447816120 wait complete. 2024-01-21T03:20:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.341+0000 7f2973290640 1 librados: shutdown 2024-01-21T03:20:09.337 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313224 got 90194313225 for osd.1 2024-01-21T03:20:09.338 DEBUG:teuthology.parallel:result is None 2024-01-21T03:20:09.338 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-21T03:20:09.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T03:20:09.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.451+0000 7f8b9117c640 10 monclient: get_monmap_and_config 2024-01-21T03:20:09.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.451+0000 7f8b9117c640 10 monclient: build_initial_monmap 2024-01-21T03:20:09.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.451+0000 7f8b9117c640 10 monclient: monmap: 2024-01-21T03:20:09.437 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:09.437 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:09.437 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:09.452599+0000 2024-01-21T03:20:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:09.452599+0000 2024-01-21T03:20:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:09.438 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:09.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 1 Processor -- start 2024-01-21T03:20:09.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 1 -- start start 2024-01-21T03:20:09.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 10 monclient: init 2024-01-21T03:20:09.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8b8c0a89d0 0x7f8b8c0a8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 10 monclient(hunting): picked mon.noname-c con 0x7f8b8c0a89d0 addr v2:172.21.15.177:3300/0 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8b8c0a9c50 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8b8c142bc0 0x7f8b8c13f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 10 monclient(hunting): picked mon.noname-b con 0x7f8b8c142bc0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8c142020 con 0x7f8b8c0a9c50 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8c0aa940 con 0x7f8b8c0a89d0 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0aac20 con 0x7f8b8c142bc0 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.452+0000 7f8b9117c640 10 monclient(hunting): authenticate will time out at 1355.768564s 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.453+0000 7f8b8b59e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8b8c142bc0 0x7f8b8c13f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.453+0000 7f8b8b59e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8b8c142bc0 0x7f8b8c13f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60200/0 (socket says 172.21.15.7:60200) 2024-01-21T03:20:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.453+0000 7f8b8b59e640 1 -- 172.21.15.7:0/160681856 learned_addr learned my addr 172.21.15.7:0/160681856 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.453+0000 7f8b8b59e640 10 monclient(hunting): get_auth_request con 0x7f8b8c142bc0 auth_method 0 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.453+0000 7f8b8b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.453+0000 7f8b8b59e640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.453+0000 7f8b8b59e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.453+0000 7f8b8ad9d640 1 --2- 172.21.15.7:0/160681856 >> v2:172.21.15.177:3300/0 conn(0x7f8b8c0a89d0 0x7f8b8c0a8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.453+0000 7f8b8ad9d640 10 monclient(hunting): get_auth_request con 0x7f8b8c0a89d0 auth_method 0 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.453+0000 7f8b8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.453+0000 7f8b8ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.453+0000 7f8b8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.454+0000 7f8b8b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.454+0000 7f8b8b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.454+0000 7f8b8b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.454+0000 7f8b8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.454+0000 7f8b8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.454+0000 7f8b8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:09.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.455+0000 7f8b8b59e640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-01-21T03:20:09.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.455+0000 7f8b8b59e640 1 -- 172.21.15.7:0/160681856 >> v2:172.21.15.177:3300/0 conn(0x7f8b8c0a89d0 msgr2=0x7f8b8c0a8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:09.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.455+0000 7f8b8b59e640 1 --2- 172.21.15.7:0/160681856 >> v2:172.21.15.177:3300/0 conn(0x7f8b8c0a89d0 0x7f8b8c0a8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.455+0000 7f8b8b59e640 1 -- 172.21.15.7:0/160681856 >> v1:172.21.15.177:6789/0 conn(0x7f8b8c0a9c50 legacy=0x7f8b8c1424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:09.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.455+0000 7f8b8b59e640 10 monclient: _finish_hunting 0 2024-01-21T03:20:09.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.455+0000 7f8b8b59e640 1 monclient: found mon.noname-b 2024-01-21T03:20:09.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.455+0000 7f8b8b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:09.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.455+0000 7f8b8b59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:09.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.455+0000 7f8b8b59e640 1 -- 172.21.15.7:0/160681856 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c0b2230 con 0x7f8b8c142bc0 2024-01-21T03:20:09.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.457+0000 7f8b8b59e640 1 --2- 172.21.15.7:0/160681856 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8b8c142bc0 0x7f8b8c13f160 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8b84009e50 tx=0x7f8b8402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10e8841bded0d717 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:09.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b89d9b640 1 -- 172.21.15.7:0/160681856 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8402c9e0 con 0x7f8b8c142bc0 2024-01-21T03:20:09.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b89d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:09.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b89d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b89d9b640 10 monclient: dump: 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b89d9b640 10 monclient: _finish_auth 0 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b89d9b640 10 monclient: _check_auth_tickets 2024-01-21T03:20:09.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b89d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b89d9b640 1 -- 172.21.15.7:0/160681856 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8402cb40 con 0x7f8b8c142bc0 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b89d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b89d9b640 1 -- 172.21.15.7:0/160681856 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b840343f0 con 0x7f8b8c142bc0 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b9117c640 5 monclient: authenticate success, global_id 4331 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b9117c640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b9117c640 4 set_mon_vals no callback set 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b89d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b89d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b89d9b640 10 monclient: dump: 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b9117c640 10 monclient: shutdown 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b9117c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b9117c640 1 -- 172.21.15.7:0/160681856 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8b8c142bc0 msgr2=0x7f8b8c13f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b9117c640 1 --2- 172.21.15.7:0/160681856 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8b8c142bc0 0x7f8b8c13f160 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8b84009e50 tx=0x7f8b8402be40 comp rx=0 tx=0).stop 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b9117c640 1 -- 172.21.15.7:0/160681856 shutdown_connections 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b9117c640 1 --2- 172.21.15.7:0/160681856 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8b8c142bc0 0x7f8b8c13f160 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b9117c640 1 --2- 172.21.15.7:0/160681856 >> v2:172.21.15.177:3300/0 conn(0x7f8b8c0a89d0 0x7f8b8c0a8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b9117c640 1 -- 172.21.15.7:0/160681856 >> 172.21.15.7:0/160681856 conn(0x7f8b8c066eb0 msgr2=0x7f8b8c0b7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b9117c640 1 -- 172.21.15.7:0/160681856 shutdown_connections 2024-01-21T03:20:09.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.461+0000 7f8b9117c640 1 -- 172.21.15.7:0/160681856 wait complete. 2024-01-21T03:20:09.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.463+0000 7f8b9117c640 10 monclient: build_initial_monmap 2024-01-21T03:20:09.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.463+0000 7f8b9117c640 10 monclient: monmap: 2024-01-21T03:20:09.454 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:09.454 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:09.454 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:09.454 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:09.454 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:09.454 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:09.454 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:09.454 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:09.454 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.463+0000 7f8b9117c640 1 librados: starting msgr at 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.463+0000 7f8b9117c640 1 librados: starting objecter 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.463+0000 7f8b9117c640 1 Processor -- start 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.463+0000 7f8b9117c640 1 -- start start 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.463+0000 7f8b9117c640 1 librados: setting wanted keys 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.463+0000 7f8b9117c640 1 librados: calling monclient init 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.463+0000 7f8b9117c640 10 monclient: init 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c0a89d0 0x7f8b8c149060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 10 monclient(hunting): picked mon.noname-b con 0x7f8b8c0a89d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8b8c0a9c50 0x7f8b8c139d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:09.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 10 monclient(hunting): picked mon.noname-c con 0x7f8b8c0a9c50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b8c142bc0 0x7f8b8c13a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 10 monclient(hunting): picked mon.noname-a con 0x7f8b8c142bc0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c14b1e0 con 0x7f8b8c142bc0 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c14b4c0 con 0x7f8b8c0a89d0 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c149db0 con 0x7f8b8c0a9c50 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b9117c640 10 monclient(hunting): authenticate will time out at 1355.780001s 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b8a59c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8b8c0a9c50 0x7f8b8c139d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b8b59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b8c142bc0 0x7f8b8c13a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b8ad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c0a89d0 0x7f8b8c149060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b8a59c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8b8c0a9c50 0x7f8b8c139d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60206/0 (socket says 172.21.15.7:60206) 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b8a59c640 1 -- 172.21.15.7:0/38059234 learned_addr learned my addr 172.21.15.7:0/38059234 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:09.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b8a59c640 10 monclient(hunting): get_auth_request con 0x7f8b8c0a9c50 auth_method 0 2024-01-21T03:20:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b8a59c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.464+0000 7f8b8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8ad9d640 10 monclient(hunting): get_auth_request con 0x7f8b8c0a89d0 auth_method 0 2024-01-21T03:20:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8b59e640 10 monclient(hunting): get_auth_request con 0x7f8b8c142bc0 auth_method 0 2024-01-21T03:20:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8b59e640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8b59e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.465+0000 7f8b8b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.468+0000 7f8b8ad9d640 10 monclient(hunting): handle_auth_done global_id 4414 payload 824 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.468+0000 7f8b8ad9d640 1 -- 172.21.15.7:0/38059234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8b8c0a9c50 msgr2=0x7f8b8c139d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.468+0000 7f8b8ad9d640 1 --2- 172.21.15.7:0/38059234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8b8c0a9c50 0x7f8b8c139d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.468+0000 7f8b8ad9d640 1 -- 172.21.15.7:0/38059234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b8c142bc0 msgr2=0x7f8b8c13a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.468+0000 7f8b8ad9d640 1 --2- 172.21.15.7:0/38059234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b8c142bc0 0x7f8b8c13a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.468+0000 7f8b8ad9d640 10 monclient: _finish_hunting 0 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.468+0000 7f8b8ad9d640 1 monclient: found mon.noname-b 2024-01-21T03:20:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.468+0000 7f8b8ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:09.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.468+0000 7f8b8ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:09.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.468+0000 7f8b8ad9d640 1 -- 172.21.15.7:0/38059234 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b84009b30 con 0x7f8b8c0a89d0 2024-01-21T03:20:09.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.470+0000 7f8b8ad9d640 1 --2- 172.21.15.7:0/38059234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c0a89d0 0x7f8b8c149060 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00e850 tx=0x7f8b7c00ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:09.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.470+0000 7f8b737fe640 10 client.?.objecter ms_handle_connect 0x7f8b8c0a89d0 2024-01-21T03:20:09.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.470+0000 7f8b737fe640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:09.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b737fe640 1 -- 172.21.15.7:0/38059234 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c0095b0 con 0x7f8b8c0a89d0 2024-01-21T03:20:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b737fe640 10 monclient: dump: 2024-01-21T03:20:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:09.479 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b737fe640 10 monclient: _finish_auth 0 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b737fe640 10 monclient: _check_auth_tickets 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b737fe640 1 -- 172.21.15.7:0/38059234 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7c00c940 con 0x7f8b8c0a89d0 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b737fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b737fe640 1 -- 172.21.15.7:0/38059234 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c00ccd0 con 0x7f8b8c0a89d0 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b737fe640 10 monclient: dump: 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b9117c640 5 monclient: authenticate success, global_id 4414 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b9117c640 10 monclient: _renew_subs 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b9117c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b9117c640 1 -- 172.21.15.7:0/38059234 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8c13a7b0 con 0x7f8b8c0a89d0 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b8959a640 4 set_mon_vals no callback set 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b9117c640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b9117c640 10 monclient: _renew_subs 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b9117c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b9117c640 1 -- 172.21.15.7:0/38059234 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8c1495a0 con 0x7f8b8c0a89d0 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.492+0000 7f8b9117c640 1 librados: init done 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.493+0000 7f8b9117c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.493+0000 7f8b9117c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:09.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.493+0000 7f8b9117c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:09.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.493+0000 7f8b9117c640 1 -- 172.21.15.7:0/38059234 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8c149f70 con 0x7f8b8c0a89d0 2024-01-21T03:20:09.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.731+0000 7f8b737fe640 1 -- 172.21.15.7:0/38059234 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8b7c013470 con 0x7f8b8c0a89d0 2024-01-21T03:20:09.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.731+0000 7f8b737fe640 1 --2- 172.21.15.7:0/38059234 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f8b7806c8e0 0x7f8b7806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:09.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.731+0000 7f8b8a59c640 1 --2- 172.21.15.7:0/38059234 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f8b7806c8e0 0x7f8b7806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:09.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.732+0000 7f8b8a59c640 10 monclient: get_auth_request con 0x7f8b7806c8e0 auth_method 0 2024-01-21T03:20:09.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.732+0000 7f8b8a59c640 1 --2- 172.21.15.7:0/38059234 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f8b7806c8e0 0x7f8b7806eda0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c148060 tx=0x7f8b80009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:09.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.732+0000 7f8b737fe640 10 client.4414.objecter ms_handle_connect 0x7f8b7806c8e0 2024-01-21T03:20:09.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.735+0000 7f8b737fe640 1 -- 172.21.15.7:0/38059234 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8b7c00ce30 con 0x7f8b8c0a89d0 2024-01-21T03:20:09.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.735+0000 7f8b737fe640 10 client.4414.objecter ms_dispatch 0x7f8b8c000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:09.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.735+0000 7f8b737fe640 3 client.4414.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:09.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.735+0000 7f8b737fe640 3 client.4414.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:09.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.735+0000 7f8b737fe640 20 client.4414.objecter dump_active .. 0 homeless 2024-01-21T03:20:09.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.747+0000 7f8b737fe640 1 -- 172.21.15.7:0/38059234 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8b7c054e70 con 0x7f8b8c0a89d0 2024-01-21T03:20:09.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.747+0000 7f8b737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:09.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.747+0000 7f8b737fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:09.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.926+0000 7f8b9117c640 1 -- 172.21.15.7:0/38059234 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8b8c14c740 con 0x7f8b7806c8e0 2024-01-21T03:20:09.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.928+0000 7f8b737fe640 1 -- 172.21.15.7:0/38059234 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48008 (secure 0 0 0) 0x7f8b8000f820 con 0x7f8b7806c8e0 2024-01-21T03:20:09.915 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:20:09.916 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-01-21T03:20:09.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 10 librados: watch_flush enter 2024-01-21T03:20:09.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 10 librados: watch_flush exit 2024-01-21T03:20:09.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 20 client.4414.objecter shutdown clearing up homeless session... 2024-01-21T03:20:09.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 10 client.4414.objecter successfully canceled tick 2024-01-21T03:20:09.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 1 -- 172.21.15.7:0/38059234 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f8b7806c8e0 msgr2=0x7f8b7806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 1 --2- 172.21.15.7:0/38059234 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f8b7806c8e0 0x7f8b7806eda0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c148060 tx=0x7f8b80009040 comp rx=0 tx=0).stop 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 10 monclient: shutdown 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 1 -- 172.21.15.7:0/38059234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c0a89d0 msgr2=0x7f8b8c149060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 1 --2- 172.21.15.7:0/38059234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c0a89d0 0x7f8b8c149060 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00e850 tx=0x7f8b7c00ed10 comp rx=0 tx=0).stop 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 1 -- 172.21.15.7:0/38059234 shutdown_connections 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 1 --2- 172.21.15.7:0/38059234 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f8b7806c8e0 0x7f8b7806eda0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 1 --2- 172.21.15.7:0/38059234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b8c142bc0 0x7f8b8c13a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 1 --2- 172.21.15.7:0/38059234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8b8c0a9c50 0x7f8b8c139d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 1 --2- 172.21.15.7:0/38059234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c0a89d0 0x7f8b8c149060 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 1 -- 172.21.15.7:0/38059234 >> 172.21.15.7:0/38059234 conn(0x7f8b8c066eb0 msgr2=0x7f8b8c1424b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 1 -- 172.21.15.7:0/38059234 shutdown_connections 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 1 -- 172.21.15.7:0/38059234 wait complete. 2024-01-21T03:20:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:09.930+0000 7f8b9117c640 1 librados: shutdown 2024-01-21T03:20:09.927 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-01-21T03:20:09.516384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218880,"kb_used_data":2704,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755840,"statfs":{"total":773094113280,"available":772869980160,"internally_reserved":0,"allocated":2768896,"data_stored":1490048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001407"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:03.669482+0000","last_change":"2024-01-21T03:20:02.233166+0000","last_active":"2024-01-21T03:20:03.669482+0000","last_peered":"2024-01-21T03:20:03.669482+0000","last_clean":"2024-01-21T03:20:03.669482+0000","last_became_active":"2024-01-21T03:20:02.226975+0000","last_became_peered":"2024-01-21T03:20:02.226975+0000","last_unstale":"2024-01-21T03:20:03.669482+0000","last_undegraded":"2024-01-21T03:20:03.669482+0000","last_fullsized":"2024-01-21T03:20:03.669482+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:36:45.013749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:02.966281+0000","last_change":"2024-01-21T03:20:02.260437+0000","last_active":"2024-01-21T03:20:02.966281+0000","last_peered":"2024-01-21T03:20:02.966281+0000","last_clean":"2024-01-21T03:20:02.966281+0000","last_became_active":"2024-01-21T03:20:02.234092+0000","last_became_peered":"2024-01-21T03:20:02.234092+0000","last_unstale":"2024-01-21T03:20:02.966281+0000","last_undegraded":"2024-01-21T03:20:02.966281+0000","last_fullsized":"2024-01-21T03:20:02.966281+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:32:42.605439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:03.664342+0000","last_change":"2024-01-21T03:20:01.878033+0000","last_active":"2024-01-21T03:20:03.664342+0000","last_peered":"2024-01-21T03:20:03.664342+0000","last_clean":"2024-01-21T03:20:03.664342+0000","last_became_active":"2024-01-21T03:20:01.872724+0000","last_became_peered":"2024-01-21T03:20:01.872724+0000","last_unstale":"2024-01-21T03:20:03.664342+0000","last_undegraded":"2024-01-21T03:20:03.664342+0000","last_fullsized":"2024-01-21T03:20:03.664342+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:06:41.515884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:02.972657+0000","last_change":"2024-01-21T03:20:02.214415+0000","last_active":"2024-01-21T03:20:02.972657+0000","last_peered":"2024-01-21T03:20:02.972657+0000","last_clean":"2024-01-21T03:20:02.972657+0000","last_became_active":"2024-01-21T03:20:02.207814+0000","last_became_peered":"2024-01-21T03:20:02.207814+0000","last_unstale":"2024-01-21T03:20:02.972657+0000","last_undegraded":"2024-01-21T03:20:02.972657+0000","last_fullsized":"2024-01-21T03:20:02.972657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:12:28.359349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:03.653205+0000","last_change":"2024-01-21T03:20:02.210632+0000","last_active":"2024-01-21T03:20:03.653205+0000","last_peered":"2024-01-21T03:20:03.653205+0000","last_clean":"2024-01-21T03:20:03.653205+0000","last_became_active":"2024-01-21T03:20:02.201314+0000","last_became_peered":"2024-01-21T03:20:02.201314+0000","last_unstale":"2024-01-21T03:20:03.653205+0000","last_undegraded":"2024-01-21T03:20:03.653205+0000","last_fullsized":"2024-01-21T03:20:03.653205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:55:53.424403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:03.637694+0000","last_change":"2024-01-21T03:20:01.846748+0000","last_active":"2024-01-21T03:20:03.637694+0000","last_peered":"2024-01-21T03:20:03.637694+0000","last_clean":"2024-01-21T03:20:03.637694+0000","last_became_active":"2024-01-21T03:20:01.818592+0000","last_became_peered":"2024-01-21T03:20:01.818592+0000","last_unstale":"2024-01-21T03:20:03.637694+0000","last_undegraded":"2024-01-21T03:20:03.637694+0000","last_fullsized":"2024-01-21T03:20:03.637694+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:08:09.861694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:03.645151+0000","last_change":"2024-01-21T03:20:01.812917+0000","last_active":"2024-01-21T03:20:03.645151+0000","last_peered":"2024-01-21T03:20:03.645151+0000","last_clean":"2024-01-21T03:20:03.645151+0000","last_became_active":"2024-01-21T03:20:01.784281+0000","last_became_peered":"2024-01-21T03:20:01.784281+0000","last_unstale":"2024-01-21T03:20:03.645151+0000","last_undegraded":"2024-01-21T03:20:03.645151+0000","last_fullsized":"2024-01-21T03:20:03.645151+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:26:56.084862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:03.659266+0000","last_change":"2024-01-21T03:20:02.295222+0000","last_active":"2024-01-21T03:20:03.659266+0000","last_peered":"2024-01-21T03:20:03.659266+0000","last_clean":"2024-01-21T03:20:03.659266+0000","last_became_active":"2024-01-21T03:20:02.289915+0000","last_became_peered":"2024-01-21T03:20:02.289915+0000","last_unstale":"2024-01-21T03:20:03.659266+0000","last_undegraded":"2024-01-21T03:20:03.659266+0000","last_fullsized":"2024-01-21T03:20:03.659266+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:36:11.210361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'76","reported_seq":35,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-21T03:20:01.556838+0000","last_change":"2024-01-21T03:19:59.052204+0000","last_active":"2024-01-21T03:20:01.556838+0000","last_peered":"2024-01-21T03:20:01.556838+0000","last_clean":"2024-01-21T03:20:01.556838+0000","last_became_active":"2024-01-21T03:19:58.060095+0000","last_became_peered":"2024-01-21T03:19:58.060095+0000","last_unstale":"2024-01-21T03:20:01.556838+0000","last_undegraded":"2024-01-21T03:20:01.556838+0000","last_fullsized":"2024-01-21T03:20:01.556838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:44.383554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:44.383554+0000","last_clean_scrub_stamp":"2024-01-21T03:19:44.383554+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:42:45.445502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280518,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":86796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.6170000000000009}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7189999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.175999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.611000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.120000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.280999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.943999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313225,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":86796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.579000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.350000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.204000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":28.372}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.126000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.155000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":32.947000000000003}]}]},{"osd":1,"up_from":21,"seq":90194313225,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":647168,"data_stored":484636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0129999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.927}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.832999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.149999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.492999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.841999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":86796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7089999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.331}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.576000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.532}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.524999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.326000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.590999999999999}]}]},{"osd":6,"up_from":18,"seq":77309411338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":86796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.965}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.121}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.468}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.178999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.3239999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.093999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.987}]}]},{"osd":5,"up_from":18,"seq":77309411339,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":647168,"data_stored":484636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5739999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9359999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.115}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.724}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.487}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.350999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.006}]}]},{"osd":7,"up_from":14,"seq":60129542153,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":86796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5890000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.968999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.542999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0749999999999993}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.125}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.134}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.260999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542153,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":86796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7050000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.891999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.670999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.283999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.98}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.628}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T03:20:09.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T03:20:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.042+0000 7f7c775e2640 10 monclient: get_monmap_and_config 2024-01-21T03:20:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.042+0000 7f7c775e2640 10 monclient: build_initial_monmap 2024-01-21T03:20:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.042+0000 7f7c775e2640 10 monclient: monmap: 2024-01-21T03:20:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:10.043595+0000 2024-01-21T03:20:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:10.043595+0000 2024-01-21T03:20:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:10.029 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.043+0000 7f7c775e2640 1 Processor -- start 2024-01-21T03:20:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.043+0000 7f7c775e2640 1 -- start start 2024-01-21T03:20:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.043+0000 7f7c775e2640 10 monclient: init 2024-01-21T03:20:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.043+0000 7f7c775e2640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.043+0000 7f7c775e2640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-01-21T03:20:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c775e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c700a8a70 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c775e2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7c700a8e50 0x7f7c70142450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c775e2640 10 monclient(hunting): picked mon.noname-b con 0x7f7c700a8e50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c775e2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c7013f6d0 0x7f7c70142990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c775e2640 10 monclient(hunting): picked mon.noname-a con 0x7f7c7013f6d0 addr v2:172.21.15.7:3300/0 2024-01-21T03:20:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c775e2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c700aa690 con 0x7f7c700a8a70 2024-01-21T03:20:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c775e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c775e2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c700a9f90 con 0x7f7c7013f6d0 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c775e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c775e2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70142ed0 con 0x7f7c700a8e50 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c775e2640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c775e2640 10 monclient(hunting): authenticate will time out at 1356.359831s 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c7537e640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f7c700a8a70 0x7f7c7013efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:36938/0 (socket says 172.21.15.7:36938) 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c7537e640 1 -- 172.21.15.7:0/2900508654 learned_addr learned my addr 172.21.15.7:0/2900508654 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c74b7d640 1 --2- 172.21.15.7:0/2900508654 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7c700a8e50 0x7f7c70142450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c75b7f640 1 --2- 172.21.15.7:0/2900508654 >> v2:172.21.15.7:3300/0 conn(0x7f7c7013f6d0 0x7f7c70142990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c74b7d640 10 monclient(hunting): get_auth_request con 0x7f7c700a8e50 auth_method 0 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c74b7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c74b7d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.044+0000 7f7c74b7d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.045+0000 7f7c75b7f640 10 monclient(hunting): get_auth_request con 0x7f7c7013f6d0 auth_method 0 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.045+0000 7f7c75b7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.045+0000 7f7c75b7f640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.045+0000 7f7c75b7f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.045+0000 7f7c74b7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.045+0000 7f7c74b7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.045+0000 7f7c74b7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.045+0000 7f7c75b7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.045+0000 7f7c75b7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.045+0000 7f7c75b7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.046+0000 7f7c67fff640 1 -- 172.21.15.7:0/2900508654 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129938110 0 0) 0x7f7c700aa690 con 0x7f7c700a8a70 2024-01-21T03:20:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.046+0000 7f7c67fff640 10 monclient(hunting): _negotiate 2024-01-21T03:20:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.046+0000 7f7c67fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.046+0000 7f7c67fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.046+0000 7f7c67fff640 10 monclient(hunting): my global_id is 4417 2024-01-21T03:20:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.046+0000 7f7c67fff640 1 -- 172.21.15.7:0/2900508654 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c5c003520 con 0x7f7c700a8a70 2024-01-21T03:20:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.047+0000 7f7c74b7d640 10 monclient(hunting): handle_auth_done global_id 4337 payload 274 2024-01-21T03:20:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.047+0000 7f7c74b7d640 1 -- 172.21.15.7:0/2900508654 >> v2:172.21.15.7:3300/0 conn(0x7f7c7013f6d0 msgr2=0x7f7c70142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:20:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.047+0000 7f7c74b7d640 1 --2- 172.21.15.7:0/2900508654 >> v2:172.21.15.7:3300/0 conn(0x7f7c7013f6d0 0x7f7c70142990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.047+0000 7f7c74b7d640 1 -- 172.21.15.7:0/2900508654 >> v1:172.21.15.177:6789/0 conn(0x7f7c700a8a70 legacy=0x7f7c7013efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:10.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.047+0000 7f7c74b7d640 10 monclient: _finish_hunting 0 2024-01-21T03:20:10.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.047+0000 7f7c74b7d640 1 monclient: found mon.noname-b 2024-01-21T03:20:10.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.047+0000 7f7c74b7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:10.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.047+0000 7f7c74b7d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:10.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.047+0000 7f7c74b7d640 1 -- 172.21.15.7:0/2900508654 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c700aa970 con 0x7f7c700a8e50 2024-01-21T03:20:10.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.049+0000 7f7c74b7d640 1 --2- 172.21.15.7:0/2900508654 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7c700a8e50 0x7f7c70142450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7c600097a0 tx=0x7f7c6002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35bf40aae760ecdc server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:10.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c67fff640 1 -- 172.21.15.7:0/2900508654 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6002e070 con 0x7f7c700a8e50 2024-01-21T03:20:10.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:10.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c67fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:10.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c67fff640 10 monclient: dump: 2024-01-21T03:20:10.038 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c67fff640 10 monclient: _finish_auth 0 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c67fff640 10 monclient: _check_auth_tickets 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c67fff640 1 -- 172.21.15.7:0/2900508654 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c600315a0 con 0x7f7c700a8e50 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c775e2640 5 monclient: authenticate success, global_id 4337 2024-01-21T03:20:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c775e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c67fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c775e2640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c775e2640 4 set_mon_vals no callback set 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c67fff640 1 -- 172.21.15.7:0/2900508654 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60031930 con 0x7f7c700a8e50 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.052+0000 7f7c67fff640 10 monclient: dump: 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:10.041 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:10.041 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:10.041 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.053+0000 7f7c775e2640 10 monclient: shutdown 2024-01-21T03:20:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.053+0000 7f7c775e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.053+0000 7f7c775e2640 1 -- 172.21.15.7:0/2900508654 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7c700a8e50 msgr2=0x7f7c70142450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.053+0000 7f7c775e2640 1 --2- 172.21.15.7:0/2900508654 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7c700a8e50 0x7f7c70142450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7c600097a0 tx=0x7f7c6002b840 comp rx=0 tx=0).stop 2024-01-21T03:20:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.053+0000 7f7c775e2640 1 -- 172.21.15.7:0/2900508654 shutdown_connections 2024-01-21T03:20:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.053+0000 7f7c775e2640 1 --2- 172.21.15.7:0/2900508654 >> v2:172.21.15.7:3300/0 conn(0x7f7c7013f6d0 0x7f7c70142990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.053+0000 7f7c775e2640 1 --2- 172.21.15.7:0/2900508654 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7c700a8e50 0x7f7c70142450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.053+0000 7f7c775e2640 1 -- 172.21.15.7:0/2900508654 >> 172.21.15.7:0/2900508654 conn(0x7f7c70066f00 msgr2=0x7f7c700b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.053+0000 7f7c775e2640 1 -- 172.21.15.7:0/2900508654 shutdown_connections 2024-01-21T03:20:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.053+0000 7f7c775e2640 1 -- 172.21.15.7:0/2900508654 wait complete. 2024-01-21T03:20:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.054+0000 7f7c775e2640 10 monclient: build_initial_monmap 2024-01-21T03:20:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.054+0000 7f7c775e2640 10 monclient: monmap: 2024-01-21T03:20:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.054+0000 7f7c775e2640 1 librados: starting msgr at 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.054+0000 7f7c775e2640 1 librados: starting objecter 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.054+0000 7f7c775e2640 1 Processor -- start 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.054+0000 7f7c775e2640 1 -- start start 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.054+0000 7f7c775e2640 1 librados: setting wanted keys 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.054+0000 7f7c775e2640 1 librados: calling monclient init 2024-01-21T03:20:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.054+0000 7f7c775e2640 10 monclient: init 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c700a8a70 0x7f7c70137470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 10 monclient(hunting): picked mon.noname-b con 0x7f7c700a8a70 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c700a8e50 0x7f7c70141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 10 monclient(hunting): picked mon.noname-a con 0x7f7c700a8e50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7c7013f6d0 0x7f7c70134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 10 monclient(hunting): picked mon.noname-c con 0x7f7c7013f6d0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70139220 con 0x7f7c700a8e50 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70136bb0 con 0x7f7c700a8a70 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70136d70 con 0x7f7c7013f6d0 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c775e2640 10 monclient(hunting): authenticate will time out at 1356.371051s 2024-01-21T03:20:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c75b7f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7c7013f6d0 0x7f7c70134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c74b7d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c700a8e50 0x7f7c70141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.055+0000 7f7c7537e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c700a8a70 0x7f7c70137470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c75b7f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7c7013f6d0 0x7f7c70134b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60224/0 (socket says 172.21.15.7:60224) 2024-01-21T03:20:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c75b7f640 1 -- 172.21.15.7:0/2326514380 learned_addr learned my addr 172.21.15.7:0/2326514380 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c75b7f640 10 monclient(hunting): get_auth_request con 0x7f7c7013f6d0 auth_method 0 2024-01-21T03:20:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c75b7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c75b7f640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c75b7f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c7537e640 10 monclient(hunting): get_auth_request con 0x7f7c700a8a70 auth_method 0 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c7537e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c7537e640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c7537e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c74b7d640 10 monclient(hunting): get_auth_request con 0x7f7c700a8e50 auth_method 0 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c74b7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c74b7d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c74b7d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c75b7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c75b7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c75b7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c7537e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c7537e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c7537e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c74b7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c74b7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.056+0000 7f7c74b7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.059+0000 7f7c7537e640 10 monclient(hunting): handle_auth_done global_id 4420 payload 824 2024-01-21T03:20:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.059+0000 7f7c7537e640 1 -- 172.21.15.7:0/2326514380 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7c7013f6d0 msgr2=0x7f7c70134b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.059+0000 7f7c7537e640 1 --2- 172.21.15.7:0/2326514380 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7c7013f6d0 0x7f7c70134b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.059+0000 7f7c7537e640 1 -- 172.21.15.7:0/2326514380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c700a8e50 msgr2=0x7f7c70141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.059+0000 7f7c7537e640 1 --2- 172.21.15.7:0/2326514380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c700a8e50 0x7f7c70141a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.059+0000 7f7c7537e640 10 monclient: _finish_hunting 0 2024-01-21T03:20:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.059+0000 7f7c7537e640 1 monclient: found mon.noname-b 2024-01-21T03:20:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.059+0000 7f7c7537e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.059+0000 7f7c7537e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:20:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.059+0000 7f7c7537e640 1 -- 172.21.15.7:0/2326514380 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c60009400 con 0x7f7c700a8a70 2024-01-21T03:20:10.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.061+0000 7f7c7537e640 1 --2- 172.21.15.7:0/2326514380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c700a8a70 0x7f7c70137470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00c430 tx=0x7f7c6c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:10.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.061+0000 7f7c65ffb640 10 client.?.objecter ms_handle_connect 0x7f7c700a8a70 2024-01-21T03:20:10.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.061+0000 7f7c65ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:10.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c65ffb640 1 -- 172.21.15.7:0/2326514380 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c005ce0 con 0x7f7c700a8a70 2024-01-21T03:20:10.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:10.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c65ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:10.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c65ffb640 10 monclient: dump: 2024-01-21T03:20:10.052 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:10.052 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:10.052 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.052 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.052 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:10.052 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:10.052 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:10.053 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:10.053 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:10.053 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:10.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c65ffb640 10 monclient: _finish_auth 0 2024-01-21T03:20:10.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c65ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:20:10.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:10.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c65ffb640 1 -- 172.21.15.7:0/2326514380 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c005e40 con 0x7f7c700a8a70 2024-01-21T03:20:10.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c65ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:10.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c65ffb640 1 -- 172.21.15.7:0/2326514380 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0072d0 con 0x7f7c700a8a70 2024-01-21T03:20:10.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:10.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c65ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:20:10.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c65ffb640 10 monclient: dump: 2024-01-21T03:20:10.053 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c775e2640 5 monclient: authenticate success, global_id 4420 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c775e2640 10 monclient: _renew_subs 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c677fe640 4 set_mon_vals no callback set 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c775e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.064+0000 7f7c775e2640 1 -- 172.21.15.7:0/2326514380 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c70138e00 con 0x7f7c700a8a70 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.065+0000 7f7c775e2640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:10.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.065+0000 7f7c775e2640 10 monclient: _renew_subs 2024-01-21T03:20:10.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.065+0000 7f7c775e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:10.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.065+0000 7f7c775e2640 1 -- 172.21.15.7:0/2326514380 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c701d6f30 con 0x7f7c700a8a70 2024-01-21T03:20:10.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.065+0000 7f7c775e2640 1 librados: init done 2024-01-21T03:20:10.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.066+0000 7f7c775e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:10.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.066+0000 7f7c775e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:10.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.066+0000 7f7c775e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:20:10.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.066+0000 7f7c775e2640 1 -- 172.21.15.7:0/2326514380 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7004e970 con 0x7f7c700a8a70 2024-01-21T03:20:10.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.083+0000 7f7c65ffb640 1 -- 172.21.15.7:0/2326514380 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f7c6c007430 con 0x7f7c700a8a70 2024-01-21T03:20:10.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.083+0000 7f7c65ffb640 1 --2- 172.21.15.7:0/2326514380 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7c5406c8b0 0x7f7c5406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:10.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.083+0000 7f7c74b7d640 1 --2- 172.21.15.7:0/2326514380 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7c5406c8b0 0x7f7c5406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:10.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.083+0000 7f7c74b7d640 10 monclient: get_auth_request con 0x7f7c5406c8b0 auth_method 0 2024-01-21T03:20:10.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.083+0000 7f7c65ffb640 10 client.4420.objecter ms_handle_connect 0x7f7c5406c8b0 2024-01-21T03:20:10.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.083+0000 7f7c74b7d640 1 --2- 172.21.15.7:0/2326514380 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7c5406c8b0 0x7f7c5406ed70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7c600097a0 tx=0x7f7c6003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:10.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.086+0000 7f7c65ffb640 1 -- 172.21.15.7:0/2326514380 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7c6c057d10 con 0x7f7c700a8a70 2024-01-21T03:20:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.086+0000 7f7c65ffb640 10 client.4420.objecter ms_dispatch 0x7f7c70000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.086+0000 7f7c65ffb640 3 client.4420.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.086+0000 7f7c65ffb640 3 client.4420.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.086+0000 7f7c65ffb640 20 client.4420.objecter dump_active .. 0 homeless 2024-01-21T03:20:10.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.320+0000 7f7c65ffb640 1 -- 172.21.15.7:0/2326514380 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f7c6c054050 con 0x7f7c700a8a70 2024-01-21T03:20:10.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.320+0000 7f7c65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:10.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.320+0000 7f7c65ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:10.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.533+0000 7f7c775e2640 1 -- 172.21.15.7:0/2326514380 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7c70138ff0 con 0x7f7c5406c8b0 2024-01-21T03:20:10.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.535+0000 7f7c65ffb640 1 -- 172.21.15.7:0/2326514380 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48008 (secure 0 0 0) 0x7f7c70138ff0 con 0x7f7c5406c8b0 2024-01-21T03:20:10.521 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:20:10.522 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-01-21T03:20:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 10 librados: watch_flush enter 2024-01-21T03:20:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 10 librados: watch_flush exit 2024-01-21T03:20:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 20 client.4420.objecter shutdown clearing up homeless session... 2024-01-21T03:20:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 10 client.4420.objecter successfully canceled tick 2024-01-21T03:20:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 1 -- 172.21.15.7:0/2326514380 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7c5406c8b0 msgr2=0x7f7c5406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 1 --2- 172.21.15.7:0/2326514380 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7c5406c8b0 0x7f7c5406ed70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7c600097a0 tx=0x7f7c6003d040 comp rx=0 tx=0).stop 2024-01-21T03:20:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 10 monclient: shutdown 2024-01-21T03:20:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 1 -- 172.21.15.7:0/2326514380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c700a8a70 msgr2=0x7f7c70137470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 1 --2- 172.21.15.7:0/2326514380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c700a8a70 0x7f7c70137470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00c430 tx=0x7f7c6c00c8f0 comp rx=0 tx=0).stop 2024-01-21T03:20:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 1 -- 172.21.15.7:0/2326514380 shutdown_connections 2024-01-21T03:20:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 1 --2- 172.21.15.7:0/2326514380 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7c5406c8b0 0x7f7c5406ed70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 1 --2- 172.21.15.7:0/2326514380 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7c7013f6d0 0x7f7c70134b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 1 --2- 172.21.15.7:0/2326514380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c700a8e50 0x7f7c70141a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 1 --2- 172.21.15.7:0/2326514380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c700a8a70 0x7f7c70137470 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 1 -- 172.21.15.7:0/2326514380 >> 172.21.15.7:0/2326514380 conn(0x7f7c70066f00 msgr2=0x7f7c70139660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 1 -- 172.21.15.7:0/2326514380 shutdown_connections 2024-01-21T03:20:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 1 -- 172.21.15.7:0/2326514380 wait complete. 2024-01-21T03:20:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.537+0000 7f7c775e2640 1 librados: shutdown 2024-01-21T03:20:10.533 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-01-21T03:20:09.516384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218880,"kb_used_data":2704,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755840,"statfs":{"total":773094113280,"available":772869980160,"internally_reserved":0,"allocated":2768896,"data_stored":1490048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001407"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:03.669482+0000","last_change":"2024-01-21T03:20:02.233166+0000","last_active":"2024-01-21T03:20:03.669482+0000","last_peered":"2024-01-21T03:20:03.669482+0000","last_clean":"2024-01-21T03:20:03.669482+0000","last_became_active":"2024-01-21T03:20:02.226975+0000","last_became_peered":"2024-01-21T03:20:02.226975+0000","last_unstale":"2024-01-21T03:20:03.669482+0000","last_undegraded":"2024-01-21T03:20:03.669482+0000","last_fullsized":"2024-01-21T03:20:03.669482+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:36:45.013749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:02.966281+0000","last_change":"2024-01-21T03:20:02.260437+0000","last_active":"2024-01-21T03:20:02.966281+0000","last_peered":"2024-01-21T03:20:02.966281+0000","last_clean":"2024-01-21T03:20:02.966281+0000","last_became_active":"2024-01-21T03:20:02.234092+0000","last_became_peered":"2024-01-21T03:20:02.234092+0000","last_unstale":"2024-01-21T03:20:02.966281+0000","last_undegraded":"2024-01-21T03:20:02.966281+0000","last_fullsized":"2024-01-21T03:20:02.966281+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:32:42.605439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:03.664342+0000","last_change":"2024-01-21T03:20:01.878033+0000","last_active":"2024-01-21T03:20:03.664342+0000","last_peered":"2024-01-21T03:20:03.664342+0000","last_clean":"2024-01-21T03:20:03.664342+0000","last_became_active":"2024-01-21T03:20:01.872724+0000","last_became_peered":"2024-01-21T03:20:01.872724+0000","last_unstale":"2024-01-21T03:20:03.664342+0000","last_undegraded":"2024-01-21T03:20:03.664342+0000","last_fullsized":"2024-01-21T03:20:03.664342+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:06:41.515884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:02.972657+0000","last_change":"2024-01-21T03:20:02.214415+0000","last_active":"2024-01-21T03:20:02.972657+0000","last_peered":"2024-01-21T03:20:02.972657+0000","last_clean":"2024-01-21T03:20:02.972657+0000","last_became_active":"2024-01-21T03:20:02.207814+0000","last_became_peered":"2024-01-21T03:20:02.207814+0000","last_unstale":"2024-01-21T03:20:02.972657+0000","last_undegraded":"2024-01-21T03:20:02.972657+0000","last_fullsized":"2024-01-21T03:20:02.972657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:12:28.359349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:03.653205+0000","last_change":"2024-01-21T03:20:02.210632+0000","last_active":"2024-01-21T03:20:03.653205+0000","last_peered":"2024-01-21T03:20:03.653205+0000","last_clean":"2024-01-21T03:20:03.653205+0000","last_became_active":"2024-01-21T03:20:02.201314+0000","last_became_peered":"2024-01-21T03:20:02.201314+0000","last_unstale":"2024-01-21T03:20:03.653205+0000","last_undegraded":"2024-01-21T03:20:03.653205+0000","last_fullsized":"2024-01-21T03:20:03.653205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:55:53.424403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:03.637694+0000","last_change":"2024-01-21T03:20:01.846748+0000","last_active":"2024-01-21T03:20:03.637694+0000","last_peered":"2024-01-21T03:20:03.637694+0000","last_clean":"2024-01-21T03:20:03.637694+0000","last_became_active":"2024-01-21T03:20:01.818592+0000","last_became_peered":"2024-01-21T03:20:01.818592+0000","last_unstale":"2024-01-21T03:20:03.637694+0000","last_undegraded":"2024-01-21T03:20:03.637694+0000","last_fullsized":"2024-01-21T03:20:03.637694+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:08:09.861694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:03.645151+0000","last_change":"2024-01-21T03:20:01.812917+0000","last_active":"2024-01-21T03:20:03.645151+0000","last_peered":"2024-01-21T03:20:03.645151+0000","last_clean":"2024-01-21T03:20:03.645151+0000","last_became_active":"2024-01-21T03:20:01.784281+0000","last_became_peered":"2024-01-21T03:20:01.784281+0000","last_unstale":"2024-01-21T03:20:03.645151+0000","last_undegraded":"2024-01-21T03:20:03.645151+0000","last_fullsized":"2024-01-21T03:20:03.645151+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:26:56.084862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T03:20:03.659266+0000","last_change":"2024-01-21T03:20:02.295222+0000","last_active":"2024-01-21T03:20:03.659266+0000","last_peered":"2024-01-21T03:20:03.659266+0000","last_clean":"2024-01-21T03:20:03.659266+0000","last_became_active":"2024-01-21T03:20:02.289915+0000","last_became_peered":"2024-01-21T03:20:02.289915+0000","last_unstale":"2024-01-21T03:20:03.659266+0000","last_undegraded":"2024-01-21T03:20:03.659266+0000","last_fullsized":"2024-01-21T03:20:03.659266+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:36:11.210361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'76","reported_seq":35,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-21T03:20:01.556838+0000","last_change":"2024-01-21T03:19:59.052204+0000","last_active":"2024-01-21T03:20:01.556838+0000","last_peered":"2024-01-21T03:20:01.556838+0000","last_clean":"2024-01-21T03:20:01.556838+0000","last_became_active":"2024-01-21T03:19:58.060095+0000","last_became_peered":"2024-01-21T03:19:58.060095+0000","last_unstale":"2024-01-21T03:20:01.556838+0000","last_undegraded":"2024-01-21T03:20:01.556838+0000","last_fullsized":"2024-01-21T03:20:01.556838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:44.383554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:44.383554+0000","last_clean_scrub_stamp":"2024-01-21T03:19:44.383554+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:42:45.445502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280518,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":86796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.6170000000000009}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7189999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.175999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.611000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.120000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.280999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.943999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313225,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":86796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.579000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.350000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.204000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":28.372}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.126000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.155000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":32.947000000000003}]}]},{"osd":1,"up_from":21,"seq":90194313225,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":647168,"data_stored":484636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0129999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.927}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.832999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.149999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.492999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.841999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":86796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7089999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.331}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.576000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.532}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.524999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.326000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.590999999999999}]}]},{"osd":6,"up_from":18,"seq":77309411338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":86796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.965}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.121}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.468}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.178999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.3239999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.093999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.987}]}]},{"osd":5,"up_from":18,"seq":77309411339,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":647168,"data_stored":484636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5739999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9359999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.115}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.724}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.487}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.350999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.006}]}]},{"osd":7,"up_from":14,"seq":60129542153,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":86796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5890000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.968999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.542999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0749999999999993}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.125}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.134}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.260999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542153,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":86796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7050000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.891999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.670999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.283999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.98}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.628}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T03:20:10.534 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-21T03:20:10.535 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-21T03:20:10.535 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-21T03:20:10.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-21T03:20:10.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.648+0000 7f34c30b4640 10 monclient: get_monmap_and_config 2024-01-21T03:20:10.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.648+0000 7f34c30b4640 10 monclient: build_initial_monmap 2024-01-21T03:20:10.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.648+0000 7f34c30b4640 10 monclient: monmap: 2024-01-21T03:20:10.634 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:10.634 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:10.635 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:20:10.649678+0000 2024-01-21T03:20:10.635 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:20:10.649678+0000 2024-01-21T03:20:10.635 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:10.635 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:10.635 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:20:10.635 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:20:10.635 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:20:10.635 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:20:10.635 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:20:10.635 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:10.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.649+0000 7f34c30b4640 1 Processor -- start 2024-01-21T03:20:10.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.649+0000 7f34c30b4640 1 -- start start 2024-01-21T03:20:10.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.649+0000 7f34c30b4640 10 monclient: init 2024-01-21T03:20:10.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c30b4640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:10.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c30b4640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-01-21T03:20:10.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c30b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34bc0a8a70 addr v1:172.21.15.7:6789/0 2024-01-21T03:20:10.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c30b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34bc0a8e50 addr v1:172.21.15.177:6789/0 2024-01-21T03:20:10.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c30b4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f34bc13f6d0 0x7f34bc14d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:10.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c30b4640 10 monclient(hunting): picked mon.noname-b con 0x7f34bc13f6d0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:10.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c30b4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34bc0aa690 con 0x7f34bc0a8a70 2024-01-21T03:20:10.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c30b4640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34bc0aa970 con 0x7f34bc0a8e50 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c30b4640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c30b4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc0a9f90 con 0x7f34bc13f6d0 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c30b4640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c30b4640 10 monclient(hunting): authenticate will time out at 1356.965834s 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c1651640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f34bc13f6d0 0x7f34bc14d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c0e50640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f34bc0a8a70 0x7f34bc13efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33790/0 (socket says 172.21.15.7:33790) 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c0e50640 1 -- 172.21.15.7:0/1385895637 learned_addr learned my addr 172.21.15.7:0/1385895637 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c1651640 10 monclient(hunting): get_auth_request con 0x7f34bc13f6d0 auth_method 0 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c1651640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c1651640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.650+0000 7f34c1651640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.651+0000 7f34c1651640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.651+0000 7f34c1651640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:10.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.651+0000 7f34c1651640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:10.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34bb7fe640 1 -- 172.21.15.7:0/1385895637 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213136048 0 0) 0x7f34bc0aa970 con 0x7f34bc0a8e50 2024-01-21T03:20:10.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34bb7fe640 10 monclient(hunting): _negotiate 2024-01-21T03:20:10.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34bb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:10.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:10.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34bb7fe640 10 monclient(hunting): my global_id is 4423 2024-01-21T03:20:10.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34bb7fe640 1 -- 172.21.15.7:0/1385895637 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ac003520 con 0x7f34bc0a8e50 2024-01-21T03:20:10.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34bb7fe640 1 -- 172.21.15.7:0/1385895637 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751346852 0 0) 0x7f34bc0aa690 con 0x7f34bc0a8a70 2024-01-21T03:20:10.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34bb7fe640 10 monclient(hunting): _negotiate 2024-01-21T03:20:10.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34bb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:10.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:10.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34bb7fe640 10 monclient(hunting): my global_id is 4362 2024-01-21T03:20:10.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34bb7fe640 1 -- 172.21.15.7:0/1385895637 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34bc0aa970 con 0x7f34bc0a8a70 2024-01-21T03:20:10.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34c1651640 10 monclient(hunting): handle_auth_done global_id 4343 payload 274 2024-01-21T03:20:10.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34c1651640 1 -- 172.21.15.7:0/1385895637 >> v1:172.21.15.177:6789/0 conn(0x7f34bc0a8e50 legacy=0x7f34bc142450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:10.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34c1651640 1 -- 172.21.15.7:0/1385895637 >> v1:172.21.15.7:6789/0 conn(0x7f34bc0a8a70 legacy=0x7f34bc13efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:10.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34c1651640 10 monclient: _finish_hunting 0 2024-01-21T03:20:10.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34c1651640 1 monclient: found mon.noname-b 2024-01-21T03:20:10.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34c1651640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:10.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34c1651640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:20:10.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.653+0000 7f34c1651640 1 -- 172.21.15.7:0/1385895637 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc0aac50 con 0x7f34bc13f6d0 2024-01-21T03:20:10.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.655+0000 7f34c1651640 1 --2- 172.21.15.7:0/1385895637 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f34bc13f6d0 0x7f34bc14d8b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f34b4009dc0 tx=0x7f34b402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87a0b801f2a08d3e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:10.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34bb7fe640 1 -- 172.21.15.7:0/1385895637 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b402cc40 con 0x7f34bc13f6d0 2024-01-21T03:20:10.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:10.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:20:10.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34bb7fe640 10 monclient: dump: 2024-01-21T03:20:10.645 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:10.645 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:10.645 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34bb7fe640 10 monclient: _finish_auth 0 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34bb7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34bb7fe640 1 -- 172.21.15.7:0/1385895637 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b402cda0 con 0x7f34bc13f6d0 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34c30b4640 5 monclient: authenticate success, global_id 4343 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34c30b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:10.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34c30b4640 10 monclient: get_monmap_and_config success 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34bb7fe640 1 -- 172.21.15.7:0/1385895637 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4034660 con 0x7f34bc13f6d0 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34c30b4640 4 set_mon_vals no callback set 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.658+0000 7f34bb7fe640 10 monclient: dump: 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:10.647 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:10.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.659+0000 7f34c30b4640 10 monclient: shutdown 2024-01-21T03:20:10.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.659+0000 7f34c30b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:10.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.659+0000 7f34c30b4640 1 -- 172.21.15.7:0/1385895637 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f34bc13f6d0 msgr2=0x7f34bc14d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:10.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.659+0000 7f34c30b4640 1 --2- 172.21.15.7:0/1385895637 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f34bc13f6d0 0x7f34bc14d8b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f34b4009dc0 tx=0x7f34b402be90 comp rx=0 tx=0).stop 2024-01-21T03:20:10.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.659+0000 7f34c30b4640 1 -- 172.21.15.7:0/1385895637 shutdown_connections 2024-01-21T03:20:10.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.659+0000 7f34c30b4640 1 --2- 172.21.15.7:0/1385895637 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f34bc13f6d0 0x7f34bc14d8b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:10.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.659+0000 7f34c30b4640 1 -- 172.21.15.7:0/1385895637 >> 172.21.15.7:0/1385895637 conn(0x7f34bc066f00 msgr2=0x7f34bc0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:10.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.659+0000 7f34c30b4640 1 -- 172.21.15.7:0/1385895637 shutdown_connections 2024-01-21T03:20:10.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.659+0000 7f34c30b4640 1 -- 172.21.15.7:0/1385895637 wait complete. 2024-01-21T03:20:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.660+0000 7f34c30b4640 10 monclient: build_initial_monmap 2024-01-21T03:20:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.660+0000 7f34c30b4640 10 monclient: monmap: 2024-01-21T03:20:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:20:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:20:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:20:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.660+0000 7f34c30b4640 1 librados: starting msgr at 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.660+0000 7f34c30b4640 1 librados: starting objecter 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.660+0000 7f34c30b4640 1 Processor -- start 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.660+0000 7f34c30b4640 1 -- start start 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.660+0000 7f34c30b4640 1 librados: setting wanted keys 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.660+0000 7f34c30b4640 1 librados: calling monclient init 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.660+0000 7f34c30b4640 10 monclient: init 2024-01-21T03:20:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 10 monclient: _reopen_session rank -1 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34bc0a8a70 0x7f34bc1d7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 10 monclient(hunting): picked mon.noname-a con 0x7f34bc0a8a70 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34bc0a8e50 0x7f34bc1d7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 10 monclient(hunting): picked mon.noname-b con 0x7f34bc0a8e50 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f34bc13f6d0 0x7f34bc0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 10 monclient(hunting): picked mon.noname-c con 0x7f34bc13f6d0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc0ad980 con 0x7f34bc0a8a70 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc1d6df0 con 0x7f34bc0a8e50 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 10 monclient(hunting): start opening mon connection 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc0adb60 con 0x7f34bc13f6d0 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 10 monclient(hunting): _renew_subs 2024-01-21T03:20:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c30b4640 10 monclient(hunting): authenticate will time out at 1356.977254s 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.661+0000 7f34c1651640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f34bc13f6d0 0x7f34bc0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c0e50640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34bc0a8a70 0x7f34bc1d7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c1651640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f34bc13f6d0 0x7f34bc0ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60232/0 (socket says 172.21.15.7:60232) 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c1651640 1 -- 172.21.15.7:0/1601166609 learned_addr learned my addr 172.21.15.7:0/1601166609 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c1651640 10 monclient(hunting): get_auth_request con 0x7f34bc13f6d0 auth_method 0 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c1651640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c1651640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c1651640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c0e50640 10 monclient(hunting): get_auth_request con 0x7f34bc0a8a70 auth_method 0 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c0e50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c0e50640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c0e50640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c1651640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c1651640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c1651640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34bbfff640 1 --2- 172.21.15.7:0/1601166609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34bc0a8e50 0x7f34bc1d7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c0e50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c0e50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.662+0000 7f34c0e50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.663+0000 7f34bbfff640 10 monclient(hunting): get_auth_request con 0x7f34bc0a8e50 auth_method 0 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.663+0000 7f34bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.663+0000 7f34bbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.663+0000 7f34bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.663+0000 7f34bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.663+0000 7f34bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.663+0000 7f34bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.665+0000 7f34c1651640 10 monclient(hunting): handle_auth_done global_id 4346 payload 824 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.665+0000 7f34c1651640 1 -- 172.21.15.7:0/1601166609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34bc0a8e50 msgr2=0x7f34bc1d7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.665+0000 7f34c1651640 1 --2- 172.21.15.7:0/1601166609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34bc0a8e50 0x7f34bc1d7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.665+0000 7f34c1651640 1 -- 172.21.15.7:0/1601166609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34bc0a8a70 msgr2=0x7f34bc1d7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.665+0000 7f34c1651640 1 --2- 172.21.15.7:0/1601166609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34bc0a8a70 0x7f34bc1d7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.665+0000 7f34c1651640 10 monclient: _finish_hunting 0 2024-01-21T03:20:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.665+0000 7f34c1651640 1 monclient: found mon.noname-c 2024-01-21T03:20:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.665+0000 7f34c1651640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:20:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.665+0000 7f34c1651640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:20:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.665+0000 7f34c1651640 1 -- 172.21.15.7:0/1601166609 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b4009a70 con 0x7f34bc13f6d0 2024-01-21T03:20:10.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.667+0000 7f34c1651640 1 --2- 172.21.15.7:0/1601166609 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f34bc13f6d0 0x7f34bc0ac0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f34b4009ee0 tx=0x7f34b4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:10.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.667+0000 7f34b97fa640 10 client.?.objecter ms_handle_connect 0x7f34bc13f6d0 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.667+0000 7f34b97fa640 10 client.?.objecter resend_mon_ops 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34b97fa640 1 -- 172.21.15.7:0/1601166609 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b402cc40 con 0x7f34bc13f6d0 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34b97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34b97fa640 10 monclient: dump: 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:10.658 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34b97fa640 10 monclient: _finish_auth 0 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34b97fa640 10 monclient: _check_auth_tickets 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34b97fa640 1 -- 172.21.15.7:0/1601166609 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b403e410 con 0x7f34bc13f6d0 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34c30b4640 5 monclient: authenticate success, global_id 4346 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34c30b4640 10 monclient: _renew_subs 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34c30b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34c30b4640 1 -- 172.21.15.7:0/1601166609 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34bc144360 con 0x7f34bc13f6d0 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34b97fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34b97fa640 1 -- 172.21.15.7:0/1601166609 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b403e780 con 0x7f34bc13f6d0 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34b97fa640 10 monclient: dump: 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:20:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34baffd640 4 set_mon_vals no callback set 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34c30b4640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34c30b4640 10 monclient: _renew_subs 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34c30b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34c30b4640 1 -- 172.21.15.7:0/1601166609 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34bc0ac7e0 con 0x7f34bc13f6d0 2024-01-21T03:20:10.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.671+0000 7f34c30b4640 1 librados: init done 2024-01-21T03:20:10.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.672+0000 7f34c30b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:10.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.672+0000 7f34c30b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:10.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.672+0000 7f34c30b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:10.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.672+0000 7f34c30b4640 1 -- 172.21.15.7:0/1601166609 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34bc1d6f30 con 0x7f34bc13f6d0 2024-01-21T03:20:10.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.690+0000 7f34b97fa640 1 -- 172.21.15.7:0/1601166609 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f34b403b050 con 0x7f34bc13f6d0 2024-01-21T03:20:10.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.691+0000 7f34b97fa640 1 --2- 172.21.15.7:0/1601166609 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f34a006c8e0 0x7f34a006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:20:10.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.691+0000 7f34c0e50640 1 --2- 172.21.15.7:0/1601166609 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f34a006c8e0 0x7f34a006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:20:10.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.691+0000 7f34c0e50640 10 monclient: get_auth_request con 0x7f34a006c8e0 auth_method 0 2024-01-21T03:20:10.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.691+0000 7f34b97fa640 10 client.4346.objecter ms_handle_connect 0x7f34a006c8e0 2024-01-21T03:20:10.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.691+0000 7f34c0e50640 1 --2- 172.21.15.7:0/1601166609 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f34a006c8e0 0x7f34a006eda0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f34b0005880 tx=0x7f34b0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:20:10.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.694+0000 7f34b97fa640 1 -- 172.21.15.7:0/1601166609 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f34b407ad40 con 0x7f34bc13f6d0 2024-01-21T03:20:10.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.694+0000 7f34b97fa640 10 client.4346.objecter ms_dispatch 0x7f34bc000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T03:20:10.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.694+0000 7f34b97fa640 3 client.4346.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:20:10.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.694+0000 7f34b97fa640 3 client.4346.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:20:10.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.694+0000 7f34b97fa640 20 client.4346.objecter dump_active .. 0 homeless 2024-01-21T03:20:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.946+0000 7f34b97fa640 1 -- 172.21.15.7:0/1601166609 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f34b4077080 con 0x7f34bc13f6d0 2024-01-21T03:20:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.946+0000 7f34b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:20:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:10.946+0000 7f34b97fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:20:11.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.169+0000 7f34c30b4640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-01-21T03:20:11.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.169+0000 7f34c30b4640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-01-21T03:20:11.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.169+0000 7f34c30b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:20:11.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.169+0000 7f34c30b4640 1 -- 172.21.15.7:0/1601166609 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f34bc04e970 con 0x7f34bc13f6d0 2024-01-21T03:20:11.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.176+0000 7f34b97fa640 1 -- 172.21.15.7:0/1601166609 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f34b4039080 con 0x7f34bc13f6d0 2024-01-21T03:20:11.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.176+0000 7f34b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-01-21T03:20:11.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.176+0000 7f34b97fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:20:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:20:11.162 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-21T03:20:11.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 10 librados: watch_flush enter 2024-01-21T03:20:11.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 10 librados: watch_flush exit 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 20 client.4346.objecter shutdown clearing up homeless session... 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 10 client.4346.objecter successfully canceled tick 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 1 -- 172.21.15.7:0/1601166609 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f34a006c8e0 msgr2=0x7f34a006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 1 --2- 172.21.15.7:0/1601166609 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f34a006c8e0 0x7f34a006eda0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f34b0005880 tx=0x7f34b0009040 comp rx=0 tx=0).stop 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 10 monclient: shutdown 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 1 -- 172.21.15.7:0/1601166609 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f34bc13f6d0 msgr2=0x7f34bc0ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 1 --2- 172.21.15.7:0/1601166609 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f34bc13f6d0 0x7f34bc0ac0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f34b4009ee0 tx=0x7f34b4004a40 comp rx=0 tx=0).stop 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 1 -- 172.21.15.7:0/1601166609 shutdown_connections 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 1 --2- 172.21.15.7:0/1601166609 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f34a006c8e0 0x7f34a006eda0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 1 --2- 172.21.15.7:0/1601166609 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f34bc13f6d0 0x7f34bc0ac0c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 1 --2- 172.21.15.7:0/1601166609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34bc0a8e50 0x7f34bc1d7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 1 --2- 172.21.15.7:0/1601166609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34bc0a8a70 0x7f34bc1d7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:20:11.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 1 -- 172.21.15.7:0/1601166609 >> 172.21.15.7:0/1601166609 conn(0x7f34bc066f00 msgr2=0x7f34bc0b2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:20:11.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 1 -- 172.21.15.7:0/1601166609 shutdown_connections 2024-01-21T03:20:11.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 1 -- 172.21.15.7:0/1601166609 wait complete. 2024-01-21T03:20:11.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:20:11.178+0000 7f34c30b4640 1 librados: shutdown 2024-01-21T03:20:11.176 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-21T03:20:11.176 INFO:teuthology.run_tasks:Running task workunit... 2024-01-21T03:20:11.189 INFO:tasks.workunit:Pulling workunits from ref 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T03:20:11.189 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-01-21T03:20:11.190 DEBUG:teuthology.orchestra.run.smithi007:> stat -- /home/ubuntu/cephtest/mnt.0 2024-01-21T03:20:11.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T03:20:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-01-21T03:20:11.204 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-01-21T03:20:11.260 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-01-21T03:20:11.260 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-01-21T03:20:11.315 INFO:tasks.workunit:timeout=6h 2024-01-21T03:20:11.315 INFO:tasks.workunit:cleanup=True 2024-01-21T03:20:11.316 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T03:20:11.371 INFO:tasks.workunit.client.0.smithi007.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-01-21T03:20:15.817 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:30.146 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:15.823 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:30.153 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:15.868 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:30.142 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:15.872 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:30.146 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:15.875 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:30.204 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:15.879 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:30.209 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:15.892 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:30.221 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:15.915 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:30.190 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:15.933 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:30.263 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:35.834 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:50.164 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:35.879 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:50.154 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:35.881 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:50.155 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:35.894 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:50.169 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:35.913 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:50.243 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:35.944 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:50.274 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:35.969 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:50.298 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:35.977 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:50.307 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:36.000 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:50.330 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:55.907 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:10.237 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:55.958 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:10.288 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:55.983 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:10.312 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:56.003 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:10.278 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:56.006 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:10.281 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:56.016 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:10.346 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:56.022 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:10.296 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:56.041 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:10.371 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:56.067 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:10.397 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:20:58.611 INFO:tasks.workunit.client.0.smithi007.stderr:Updating files: 96% (11707/12104) Updating files: 97% (11741/12104) Updating files: 98% (11862/12104) Updating files: 99% (11983/12104) Updating files: 100% (12104/12104) Updating files: 100% (12104/12104), done. 2024-01-21T03:20:59.363 INFO:tasks.workunit.client.0.smithi007.stderr:Note: switching to '0edf41a622739843d6b978b179ff3227b476dd9d'. 2024-01-21T03:20:59.363 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr:changes and commit them, and you can discard any commits you make in this 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr:state without impacting any branches by switching back to a branch. 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr:If you want to create a new branch to retain commits you create, you may 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr:do so (now or later) by using -c with the switch command. Example: 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr: git switch -c 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr:Or undo this operation with: 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr: git switch - 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-01-21T03:20:59.364 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-01-21T03:20:59.365 INFO:tasks.workunit.client.0.smithi007.stderr:HEAD is now at 0edf41a6227 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2024-01-18-0823 2024-01-21T03:20:59.368 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-01-21T03:20:59.433 INFO:tasks.workunit.client.0.smithi007.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-01-21T03:20:59.435 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-21T03:20:59.435 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-01-21T03:21:00.113 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-01-21T03:21:00.182 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-01-21T03:21:00.220 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-21T03:21:00.222 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-21T03:21:00.260 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-01-21T03:21:00.261 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-21T03:21:00.264 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-01-21T03:21:00.264 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-01-21T03:21:00.319 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-01-21T03:21:00.320 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-01-21T03:21:00.321 DEBUG:teuthology.orchestra.run.smithi007:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0edf41a622739843d6b978b179ff3227b476dd9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:+ parallel=1 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' '' = --serial ']' 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:+ crimson=0 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' '' = --crimson ']' 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:+ color= 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -t 1 ']' 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:+ GTEST_OUTPUT=xml:/home/ubuntu/cephtest/archive/gtest_xml_report 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:+ declare -A pids 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_aio 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_aio' 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_aio 2024-01-21T03:21:00.379 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.380 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_aio 2024-01-21T03:21:00.380 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40106 2024-01-21T03:21:00.380 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_aio on pid 40106' 2024-01-21T03:21:00.381 INFO:tasks.workunit.client.0.smithi007.stdout:test api_aio on pid 40106 2024-01-21T03:21:00.383 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40106 2024-01-21T03:21:00.383 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.383 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-01-21T03:21:00.384 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.384 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_aio_pp 2024-01-21T03:21:00.384 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_aio_pp' 2024-01-21T03:21:00.384 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_aio_pp 2024-01-21T03:21:00.384 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.384 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.384 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.384 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.384 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.384 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.384 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.384 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.384 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.385 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.386 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_aio_pp 2024-01-21T03:21:00.386 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40113 2024-01-21T03:21:00.386 INFO:tasks.workunit.client.0.smithi007.stdout:test api_aio_pp on pid 40113 2024-01-21T03:21:00.390 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_aio_pp on pid 40113' 2024-01-21T03:21:00.390 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40113 2024-01-21T03:21:00.390 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.390 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.390 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_io 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_io' 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_io 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.391 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.392 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.392 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.392 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.392 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.392 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.392 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.392 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.392 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.392 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.392 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.392 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.392 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.392 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.393 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.394 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.394 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.394 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.394 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.394 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.394 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.394 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.394 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_io 2024-01-21T03:21:00.394 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40122 2024-01-21T03:21:00.394 INFO:tasks.workunit.client.0.smithi007.stdout:test api_io on pid 40122 2024-01-21T03:21:00.399 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_io on pid 40122' 2024-01-21T03:21:00.399 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40122 2024-01-21T03:21:00.399 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.399 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.399 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-01-21T03:21:00.399 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_io_pp 2024-01-21T03:21:00.399 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_io_pp' 2024-01-21T03:21:00.399 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_io_pp 2024-01-21T03:21:00.399 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.399 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.399 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.399 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.399 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.400 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.401 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.402 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.403 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.403 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.403 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.403 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.403 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.403 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.403 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_io_pp 2024-01-21T03:21:00.403 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40134 2024-01-21T03:21:00.403 INFO:tasks.workunit.client.0.smithi007.stdout:test api_io_pp on pid 40134 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_io_pp on pid 40134' 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40134 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_asio 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_asio' 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_asio 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.410 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.411 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.412 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.413 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.414 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_asio 2024-01-21T03:21:00.415 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40149 2024-01-21T03:21:00.416 INFO:tasks.workunit.client.0.smithi007.stdout:test api_asio on pid 40149 2024-01-21T03:21:00.423 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_asio on pid 40149' 2024-01-21T03:21:00.424 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40149 2024-01-21T03:21:00.424 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.424 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.424 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-01-21T03:21:00.424 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_list 2024-01-21T03:21:00.424 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.424 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.424 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.424 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_list' 2024-01-21T03:21:00.424 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.424 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.424 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.424 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_list 2024-01-21T03:21:00.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40106/exe 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.426 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.430 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.431 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.431 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.431 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.431 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.431 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.431 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_list 2024-01-21T03:21:00.431 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.431 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40165 2024-01-21T03:21:00.431 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_list on pid 40165' 2024-01-21T03:21:00.431 INFO:tasks.workunit.client.0.smithi007.stdout:test api_list on pid 40165 2024-01-21T03:21:00.440 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40165 2024-01-21T03:21:00.440 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.440 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.440 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.440 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.440 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-01-21T03:21:00.440 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.440 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_lock 2024-01-21T03:21:00.440 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.440 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_aio.log 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_lock' 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.441 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_lock 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_aio: /' 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.442 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.443 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.444 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.445 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.446 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.447 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.448 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.448 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.448 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.448 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.448 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_lock 2024-01-21T03:21:00.448 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40186 2024-01-21T03:21:00.448 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40113/exe 2024-01-21T03:21:00.448 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_lock on pid 40186' 2024-01-21T03:21:00.448 INFO:tasks.workunit.client.0.smithi007.stdout:test api_lock on pid 40186 2024-01-21T03:21:00.457 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-01-21T03:21:00.457 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40186 2024-01-21T03:21:00.457 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.457 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.457 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_lock_pp 2024-01-21T03:21:00.457 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_lock_pp' 2024-01-21T03:21:00.457 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.457 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.458 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.459 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40122/exe 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.460 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.461 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_lock_pp 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.462 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-01-21T03:21:00.463 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.464 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_aio_pp: /' 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_lock_pp 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40206 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.465 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_lock_pp on pid 40206' 2024-01-21T03:21:00.466 INFO:tasks.workunit.client.0.smithi007.stdout:test api_lock_pp on pid 40206 2024-01-21T03:21:00.473 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40206 2024-01-21T03:21:00.473 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.473 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.473 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-01-21T03:21:00.473 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.473 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.473 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_misc 2024-01-21T03:21:00.473 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_misc' 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_io.log 2024-01-21T03:21:00.474 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_misc 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.475 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.476 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.477 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.478 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_misc 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.479 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.480 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40222 2024-01-21T03:21:00.480 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.480 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.480 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_misc on pid 40222' 2024-01-21T03:21:00.480 INFO:tasks.workunit.client.0.smithi007.stdout:test api_misc on pid 40222 2024-01-21T03:21:00.493 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.493 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40222 2024-01-21T03:21:00.493 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.493 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.493 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.493 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.493 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_io: /' 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.494 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_misc_pp 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_misc_pp' 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.495 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40134/exe 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.496 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.497 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.498 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40149/exe 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_misc_pp 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.499 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.500 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.501 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.502 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_asio: /' 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.503 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_asio.log 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_misc_pp 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.504 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40253 2024-01-21T03:21:00.505 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_misc_pp on pid 40253' 2024-01-21T03:21:00.505 INFO:tasks.workunit.client.0.smithi007.stdout:test api_misc_pp on pid 40253 2024-01-21T03:21:00.509 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40253 2024-01-21T03:21:00.509 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.509 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.509 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_tier_pp 2024-01-21T03:21:00.509 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_tier_pp' 2024-01-21T03:21:00.509 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.509 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.509 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.510 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_tier_pp 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.511 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.512 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.512 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.512 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.512 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_tier_pp 2024-01-21T03:21:00.512 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40263 2024-01-21T03:21:00.512 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_tier_pp on pid 40263' 2024-01-21T03:21:00.512 INFO:tasks.workunit.client.0.smithi007.stdout:test api_tier_pp on pid 40263 2024-01-21T03:21:00.528 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40263 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40165/exe 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_pool 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.529 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_pool' 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.530 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.531 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40186/exe 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.532 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_list.log 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.533 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_lock: /' 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.534 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_list: /' 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_io_pp: /' 2024-01-21T03:21:00.535 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.536 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.537 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.538 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.539 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_lock.log 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.540 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_pool 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.541 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.542 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.542 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.542 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.542 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_pool 2024-01-21T03:21:00.542 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.542 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40297 2024-01-21T03:21:00.543 INFO:tasks.workunit.client.0.smithi007.stdout:test api_pool on pid 40297 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_pool on pid 40297' 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40297 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.554 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.555 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_snapshots 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_snapshots' 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.556 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.557 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_snapshots 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.558 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.559 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.560 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.561 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.562 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.563 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_snapshots 2024-01-21T03:21:00.563 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40341 2024-01-21T03:21:00.563 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_snapshots on pid 40341' 2024-01-21T03:21:00.563 INFO:tasks.workunit.client.0.smithi007.stdout:test api_snapshots on pid 40341 2024-01-21T03:21:00.573 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40341 2024-01-21T03:21:00.573 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.573 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.573 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.573 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.573 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.573 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.573 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.573 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.573 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.573 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.574 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.575 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_snapshots_pp 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.576 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_snapshots_pp' 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40263/exe 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_snapshots_pp 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.577 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40206/exe 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.578 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_tier_pp: /' 2024-01-21T03:21:00.579 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40222/exe 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_snapshots_pp 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2024-01-21T03:21:00.580 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40372 2024-01-21T03:21:00.581 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_snapshots_pp on pid 40372' 2024-01-21T03:21:00.581 INFO:tasks.workunit.client.0.smithi007.stdout:test api_snapshots_pp on pid 40372 2024-01-21T03:21:00.588 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40372 2024-01-21T03:21:00.588 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.588 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-01-21T03:21:00.588 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.588 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.588 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.588 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.588 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.588 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.588 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.588 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.588 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.588 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_stat 2024-01-21T03:21:00.589 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_stat' 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_stat 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.590 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_misc: /' 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.591 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_lock_pp: /' 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.592 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.593 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_stat 2024-01-21T03:21:00.594 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.595 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.595 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.595 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.595 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.595 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.595 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40401 2024-01-21T03:21:00.595 INFO:tasks.workunit.client.0.smithi007.stdout:test api_stat on pid 40401 2024-01-21T03:21:00.602 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-01-21T03:21:00.602 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_stat on pid 40401' 2024-01-21T03:21:00.602 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40401 2024-01-21T03:21:00.602 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.602 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.602 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.602 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.602 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.602 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.602 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_misc.log 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.603 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.604 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.605 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_stat_pp 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_stat_pp' 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.606 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.607 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.607 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.607 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.607 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.607 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.607 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.607 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.607 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.607 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.607 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.607 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.607 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.607 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.608 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-01-21T03:21:00.608 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [----------] Global test environment set-up. 2024-01-21T03:21:00.608 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [----------] 12 tests from AsioRados 2024-01-21T03:21:00.608 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.493+0000 7f5885c4f780 10 monclient: get_monmap_and_config 2024-01-21T03:21:00.608 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.493+0000 7f5885c4f780 10 monclient: build_initial_monmap 2024-01-21T03:21:00.608 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.493+0000 7f5885c4f780 10 monclient: monmap: 2024-01-21T03:21:00.608 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 0 2024-01-21T03:21:00.608 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:21:00.608 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 0.000000 2024-01-21T03:21:00.608 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 0.000000 2024-01-21T03:21:00.608 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 0 (unknown) 2024-01-21T03:21:00.608 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.493+0000 7f5885c4f780 1 Processor -- start 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.493+0000 7f5885c4f780 1 -- start start 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.493+0000 7f5885c4f780 10 monclient: init 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 10 monclient: _reopen_session rank -1 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558e7efa5c30 0x558e7efa8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 10 monclient(hunting): picked mon.noname-a con 0x558e7efa5c30 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e7f03d5c0 0x558e7f03aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 10 monclient(hunting): picked mon.noname-b con 0x558e7f03d5c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x558e7ef6dae0 0x558e7f039df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:21:00.609 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 10 monclient(hunting): picked mon.noname-c con 0x558e7ef6dae0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:21:00.610 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 10 monclient(hunting): start opening mon connection 2024-01-21T03:21:00.610 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7ef26970 con 0x558e7efa5c30 2024-01-21T03:21:00.610 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 10 monclient(hunting): start opening mon connection 2024-01-21T03:21:00.610 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7eebfda0 con 0x558e7f03d5c0 2024-01-21T03:21:00.610 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 10 monclient(hunting): start opening mon connection 2024-01-21T03:21:00.610 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e7eebfa40 con 0x558e7ef6dae0 2024-01-21T03:21:00.610 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 10 monclient(hunting): _renew_subs 2024-01-21T03:21:00.610 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.494+0000 7f5885c4f780 10 monclient(hunting): authenticate will time out at 1406.811413s 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_stat_pp 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.613 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.614 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.614 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.614 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.614 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.614 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.614 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.614 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.614 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.614 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40297/exe 2024-01-21T03:21:00.614 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.614 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_stat_pp 2024-01-21T03:21:00.614 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40434 2024-01-21T03:21:00.614 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_stat_pp on pid 40434' 2024-01-21T03:21:00.615 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21test api_stat_pp on pid 40434 2024-01-21T03:21:00.615 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40434 2024-01-21T03:21:00.615 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.615 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.616 INFO:tasks.workunit.client.0.smithi007.stdout::00.501+0000 7f588381c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x558e7ef6dae0 0x558e7f039df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:21:00.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.503+0000 7f588301b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558e7efa5c30 0x558e7efa8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:21:00.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.503+0000 7f588301b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558e7efa5c30 0x558e7efa8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40902/0 (socket says 172.21.15.7:40902) 2024-01-21T03:21:00.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.503+0000 7f588301b640 1 -- 172.21.15.7:0/3748120239 learned_addr learned my addr 172.21.15.7:0/3748120239 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:21:00.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.503+0000 7f588301b640 10 monclient(hunting): get_auth_request con 0x558e7efa5c30 auth_method 0 2024-01-21T03:21:00.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.503+0000 7f588301b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:21:00.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.503+0000 7f588301b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:21:00.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.503+0000 7f588301b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:21:00.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.504+0000 7f588301b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:21:00.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.504+0000 7f588301b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:21:00.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.504+0000 7f588301b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:21:00.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.504+0000 7f588381c640 10 monclient(hunting): get_auth_request con 0x558e7ef6dae0 auth_method 0 2024-01-21T03:21:00.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.504+0000 7f588381c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:21:00.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.504+0000 7f588381c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:21:00.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.504+0000 7f588381c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:21:00.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.506+0000 7f588301b640 10 monclient(hunting): handle_auth_done global_id 4380 payload 274 2024-01-21T03:21:00.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.506+0000 7f588301b640 1 -- 172.21.15.7:0/3748120239 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x558e7ef6dae0 msgr2=0x558e7f039df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:21:00.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.506+0000 7f588301b640 1 --2- 172.21.15.7:0/3748120239 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x558e7ef6dae0 0x558e7f039df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:21:00.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.506+0000 7f588301b640 1 -- 172.21.15.7:0/3748120239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e7f03d5c0 msgr2=0x558e7f03aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:21:00.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.506+0000 7f588301b640 1 --2- 172.21.15.7:0/3748120239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e7f03d5c0 0x558e7f03aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:21:00.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.506+0000 7f588301b640 10 monclient: _finish_hunting 0 2024-01-21T03:21:00.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.506+0000 7f588301b640 1 monclient: found mon.noname-a 2024-01-21T03:21:00.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.506+0000 7f588301b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:21:00.625 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-01-21T03:21:00.625 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.625 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.625 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.625 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.625 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.625 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.625 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.625 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.625 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.625 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.626 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_pool: /' 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_pool.log 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_watch_notify 2024-01-21T03:21:00.627 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_watch_notify' 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_watch_notify 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40253/exe 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.628 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.629 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_misc_pp: /' 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_watch_notify 2024-01-21T03:21:00.630 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-01-21T03:21:00.631 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40468 2024-01-21T03:21:00.631 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00test api_watch_notify on pid 40468 2024-01-21T03:21:00.636 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_watch_notify on pid 40468' 2024-01-21T03:21:00.636 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40468 2024-01-21T03:21:00.636 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.636 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.636 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2024-01-21T03:21:00.636 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.636 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.636 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.636 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.637 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.638 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.639 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.640 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.640 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.640 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.640 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_watch_notify_pp 2024-01-21T03:21:00.640 INFO:tasks.workunit.client.0.smithi007.stdout:.506+0000 7f588301b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:21:00.640 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.506+0000 7f588301b640 1 -- 172.21.15.7:0/3748120239 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5878005de0 con 0x558e7efa5c30 2024-01-21T03:21:00.640 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.510+0000 7f588301b640 1 --2- 172.21.15.7:0/3748120239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558e7efa5c30 0x558e7efa8cc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f587400a420 tx=0x7f587400a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4107692096f14e32 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:21:00.640 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.515+0000 7f5881818640 1 -- 172.21.15.7:0/3748120239 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587400c8f0 con 0x558e7efa5c30 2024-01-21T03:21:00.640 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.515+0000 7f5881818640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:21:00.640 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.515+0000 7f5881818640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.515+0000 7f5881818640 10 monclient: dump: 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 1 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 19 (squid) 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.515+0000 7f5881818640 10 monclient: _finish_auth 0 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.515+0000 7f5881818640 10 monclient: _check_auth_tickets 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.515+0000 7f5881818640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.515+0000 7f5881818640 1 -- 172.21.15.7:0/3748120239 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587400ca50 con 0x558e7efa5c30 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.515+0000 7f5881818640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.515+0000 7f5881818640 1 -- 172.21.15.7:0/3748120239 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587400ce00 con 0x558e7efa5c30 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.515+0000 7f5881818640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.515+0000 7f5881818640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.515+0000 7f5881818640 10 monclient: dump: 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 1 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 19 (squid) 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:21:00.642 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.643 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.516+0000 7f5885c4f780 5 monclient: authenticate success, global_id 4380 2024-01-21T03:21:00.643 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.516+0000 7f5885c4f780 10 monclient: get_monmap_and_config success 2024-01-21T03:21:00.643 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.516+0000 7f5885c4f780 4 set_mon_vals no callback set 2024-01-21T03:21:00.645 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.646 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_watch_notify_pp' 2024-01-21T03:21:00.646 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.646 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.646 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.646 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.646 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.646 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.647 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.648 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.648 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.648 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.648 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.648 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.648 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.529+0000 7f5885c4f780 10 monclient: shutdown 2024-01-21T03:21:00.648 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.529+0000 7f5885c4f780 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:21:00.648 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.529+0000 7f5885c4f780 1 -- 172.21.15.7:0/3748120239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558e7efa5c30 msgr2=0x558e7efa8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:21:00.648 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.529+0000 7f5885c4f780 1 --2- 172.21.15.7:0/3748120239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558e7efa5c30 0x558e7efa8cc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f587400a420 tx=0x7f587400a8e0 comp rx=0 tx=0).stop 2024-01-21T03:21:00.648 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.530+0000 7f5885c4f780 1 -- 172.21.15.7:0/3748120239 shutdown_connections 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.530+0000 7f5885c4f780 1 --2- 172.21.15.7:0/3748120239 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x558e7ef6dae0 0x558e7f039df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.530+0000 7f5885c4f780 1 --2- 172.21.15.7:0/3748120239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e7f03d5c0 0x558e7f03aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.530+0000 7f5885c4f780 1 --2- 172.21.15.7:0/3748120239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558e7efa5c30 0x558e7efa8cc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.530+0000 7f5885c4f780 1 -- 172.21.15.7:0/3748120239 >> 172.21.15.7:0/3748120239 conn(0x558e7ef6a520 msgr2=0x558e7f03c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.530+0000 7f5885c4f780 1 -- 172.21.15.7:0/3748120239 shutdown_connections 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.530+0000 7f5885c4f780 1 -- 172.21.15.7:0/3748120239 wait complete. 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.531+0000 7f5885c4f780 10 monclient: build_initial_monmap 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.531+0000 7f5885c4f780 10 monclient: monmap: 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 0 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 0.000000 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 0.000000 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 0 (unknown) 2024-01-21T03:21:00.649 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.531+0000 7f5885c4f780 1 librados: starting msgr at 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.531+0000 7f5885c4f780 1 librados: starting objecter 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.531+0000 7f5885c4f780 1 Processor -- start 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.531+0000 7f5885c4f780 1 -- start start 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.531+0000 7f5885c4f780 1 librados: setting wanted keys 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.531+0000 7f5885c4f780 1 librados: calling monclient init 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.531+0000 7f5885c4f780 10 monclient: init 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.532+0000 7f5885c4f780 10 monclient: _reopen_session rank -1 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.532+0000 7f5885c4f780 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:21:00.650 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.532+0000 7f5885c4f780 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e7ef6dae0 0x558e7f042d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:21:00.659 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.659 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.659 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.659 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.659 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.659 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.659 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.659 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.659 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.660 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.661 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40372/exe 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.662 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.663 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_watch_notify_pp 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.664 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.665 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.666 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.666 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.666 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.666 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.666 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.666 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.666 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-01-21T03:21:00.666 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.666 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_watch_notify_pp 2024-01-21T03:21:00.666 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40522 2024-01-21T03:21:00.666 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.533+0000 7f5885c4f780test api_watch_notify_pp on pid 40522 2024-01-21T03:21:00.670 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_watch_notify_pp on pid 40522' 2024-01-21T03:21:00.670 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40522 2024-01-21T03:21:00.670 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.670 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.670 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_cmd 2024-01-21T03:21:00.670 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_cmd' 2024-01-21T03:21:00.670 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_cmd 2024-01-21T03:21:00.670 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40401/exe 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.671 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_stat.log 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.672 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.673 INFO:tasks.workunit.client.0.smithi007.stdout: 10 monclient(hunting): picked mon.noname-b con 0x558e7ef6dae0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:21:00.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.533+0000 7f5885c4f780 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x558e7efa5c30 0x558e7f034ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:21:00.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.533+0000 7f5885c4f780 10 monclient(hunting): picked mon.noname-c con 0x558e7efa5c30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:21:00.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.533+0000 7f5885c4f780 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558e7f03d5c0 0x558e7f0dbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:21:00.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.533+0000 7f5885c4f780 10 monclient(hunting): picked mon.noname-a con 0x558e7f03d5c0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:21:00.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.533+0000 7f5885c4f780 10 monclient(hunting): start opening mon connection 2024-01-21T03:21:00.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.533+0000 7f5885c4f780 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7ee9f550 con 0x558e7f03d5c0 2024-01-21T03:21:00.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.533+0000 7f5885c4f780 10 monclient(hunting): start opening mon connection 2024-01-21T03:21:00.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.533+0000 7f5885c4f780 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7f037640 con 0x558e7ef6dae0 2024-01-21T03:21:00.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.533+0000 7f5885c4f780 10 monclient(hunting): start opening mon connection 2024-01-21T03:21:00.673 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.533+0000 7f5885c4f780 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e7f03a700 con 0x558e7efa5c30 2024-01-21T03:21:00.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.533+0000 7f5885c4f780 10 monclient(hunting): _renew_subs 2024-01-21T03:21:00.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.533+0000 7f5885c4f780 10 monclient(hunting): authenticate will time out at 1406.849850s 2024-01-21T03:21:00.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.538+0000 7f588381c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558e7f03d5c0 0x558e7f0dbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:21:00.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.538+0000 7f588381c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558e7f03d5c0 0x558e7f0dbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40932/0 (socket says 172.21.15.7:40932) 2024-01-21T03:21:00.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.538+0000 7f588381c640 1 -- 172.21.15.7:0/2058430812 learned_addr learned my addr 172.21.15.7:0/2058430812 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:21:00.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.538+0000 7f588381c640 10 monclient(hunting): get_auth_request con 0x558e7f03d5c0 auth_method 0 2024-01-21T03:21:00.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.538+0000 7f588381c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:21:00.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.538+0000 7f588381c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:21:00.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.538+0000 7f588381c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:21:00.674 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.540+0000 7f588301b640 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e7ef6dae0 0x558e7f042d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:21:00.677 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.677 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.677 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.677 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.677 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.677 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.677 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.677 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.677 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_cmd 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40552 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.678 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.679 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.679 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.679 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_cmd on pid 40552' 2024-01-21T03:21:00.679 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.543+0000 7f588281a640 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x558e7efa5c30 0x558e7f034ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=test api_cmd on pid 40552 2024-01-21T03:21:00.688 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40552 2024-01-21T03:21:00.688 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40341/exe 2024-01-21T03:21:00.688 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.688 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.688 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_stat: /' 2024-01-21T03:21:00.688 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.688 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.688 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.688 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.688 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_cmd_pp 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_cmd_pp' 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.689 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.690 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.691 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.692 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_snapshots: /' 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_cmd_pp 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.693 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.694 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_cmd_pp 2024-01-21T03:21:00.695 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40579 2024-01-21T03:21:00.695 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_cmd_pp on pid 40579' 2024-01-21T03:21:00.695 INFO:tasks.workunit.client.0.smithi007.stdout:test api_cmd_pp on pid 40579 2024-01-21T03:21:00.707 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40579 2024-01-21T03:21:00.707 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.707 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.708 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.708 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.708 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.708 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.708 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.708 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.708 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.708 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.708 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.708 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.708 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.708 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.708 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_service 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_service' 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.709 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.710 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_service 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.711 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.712 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.713 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.714 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.715 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.716 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.717 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.717 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.717 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.717 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.717 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_service 2024-01-21T03:21:00.717 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40610 2024-01-21T03:21:00.717 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_service on pid 40610' 2024-01-21T03:21:00.717 INFO:tasks.workunit.client.0.smithi007.stdout:test api_service on pid 40610 2024-01-21T03:21:00.732 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40610 2024-01-21T03:21:00.732 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.732 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.732 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.732 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.732 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.732 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.732 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.732 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.732 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.732 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-01-21T03:21:00.733 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_service_pp 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_service_pp' 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.734 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_service_pp 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.735 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40434/exe 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40552/exe 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.736 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.737 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.738 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.739 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.740 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.741 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.742 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.743 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.744 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.744 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.744 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_cmd.log 2024-01-21T03:21:00.744 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.744 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.744 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.744 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_service_pp 2024-01-21T03:21:00.744 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40646 2024-01-21T03:21:00.744 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_service_pp on pid 40646' 2024-01-21T03:21:00.744 INFO:tasks.workunit.client.0.smithi007.stdout:test api_service_pp on pid 40646 2024-01-21T03:21:00.753 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40646 2024-01-21T03:21:00.753 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.753 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.753 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2024-01-21T03:21:00.753 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_cmd: /' 2024-01-21T03:21:00.753 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.753 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.754 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.755 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.756 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.757 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_c_write_operations 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_c_write_operations' 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_stat_pp: /' 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_c_write_operations 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.758 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.759 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_c_write_operations 2024-01-21T03:21:00.760 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.760 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.760 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.760 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.760 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.760 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.760 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40694 2024-01-21T03:21:00.760 INFO:tasks.workunit.client.0.smithi007.stdout:test api_c_write_operations on pid 40694 2024-01-21T03:21:00.772 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_c_write_operations on pid 40694' 2024-01-21T03:21:00.772 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40694 2024-01-21T03:21:00.772 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.772 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.772 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.772 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.772 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_c_read_operations 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40522/exe 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.773 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_c_read_operations' 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40468/exe 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.774 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.775 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.776 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_c_read_operations 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.777 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40579/exe 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.778 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.779 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.780 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.781 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.782 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.782 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.782 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_c_read_operations 2024-01-21T03:21:00.782 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40734 2024-01-21T03:21:00.782 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_c_read_operations on pid 40734' 2024-01-21T03:21:00.782 INFO:tasks.workunit.client.0.smithi007.stdout:test api_c_read_operations on pid 40734 2024-01-21T03:21:00.791 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40734 2024-01-21T03:21:00.791 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.791 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.791 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.791 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.791 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.791 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-01-21T03:21:00.791 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.791 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.791 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.791 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.792 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_watch_notify: /' 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_cls_remote_reads 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_cls_remote_reads' 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.793 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.794 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.795 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_cls_remote_reads 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_cls_remote_reads 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40751 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.796 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_cls_remote_reads on pid 40751' 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.797 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.798 INFO:tasks.workunit.client.0.smithi007.stdout:test api_cls_remote_reads on pid 40751 2024-01-21T03:21:00.805 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40751 2024-01-21T03:21:00.805 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.806 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.807 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s list_parallel 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.808 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' list_parallel' 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40610/exe 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.809 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo list_parallel 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.810 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_service: /' 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.811 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.812 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.812 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.812 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=list_parallel 2024-01-21T03:21:00.812 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40782 2024-01-21T03:21:00.812 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test list_parallel on pid 40782' 2024-01-21T03:21:00.812 INFO:tasks.workunit.client.0.smithi007.stdout:test list_parallel on pid 40782 2024-01-21T03:21:00.819 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40782 2024-01-21T03:21:00.819 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s open_pools_parallel 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.820 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' open_pools_parallel' 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_service.log 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.822 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.823 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40646/exe 2024-01-21T03:21:00.824 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo open_pools_parallel 2024-01-21T03:21:00.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.825 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.825 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.825 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.825 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.825 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.825 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=open_pools_parallel 2024-01-21T03:21:00.825 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40807 2024-01-21T03:21:00.825 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test open_pools_parallel on pid 40807' 2024-01-21T03:21:00.825 INFO:tasks.workunit.client.0.smithi007.stdout:test open_pools_parallel on pid 40807 2024-01-21T03:21:00.841 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40807 2024-01-21T03:21:00.841 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T03:21:00.841 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.842 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s delete_pools_parallel 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.843 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' delete_pools_parallel' 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.844 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_service_pp: /' 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo delete_pools_parallel 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.845 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.846 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.847 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.848 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.849 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.850 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.851 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.852 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.853 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40751/exe 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.854 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.855 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.855 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.855 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.855 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.855 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.855 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.855 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.855 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.855 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.855 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.855 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.855 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.855 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.856 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.856 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.856 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.856 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.856 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.856 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.856 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.856 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.856 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.856 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.856 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.856 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.857 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.857 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.857 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40694/exe 2024-01-21T03:21:00.857 INFO:tasks.workunit.client.0.smithi007.stdout:test delete_pools_parallel on pid 40846 2024-01-21T03:21:00.864 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=delete_pools_parallel 2024-01-21T03:21:00.864 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40846 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test delete_pools_parallel on pid 40846' 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40846 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.865 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.866 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.867 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s cls 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' cls' 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.868 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo cls 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.869 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.870 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=cls 2024-01-21T03:21:00.871 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.871 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.871 INFO:tasks.workunit.client.0.smithi007.stdout:test cls on pid 40883 2024-01-21T03:21:00.882 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.882 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.882 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40883 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test cls on pid 40883' 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40883 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.883 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.884 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.885 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s cmd 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' cmd' 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.886 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40734/exe 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.887 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo cmd 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.888 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.889 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.890 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.891 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.892 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40807/exe 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=cmd 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.893 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40920 2024-01-21T03:21:00.894 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test cmd on pid 40920' 2024-01-21T03:21:00.894 INFO:tasks.workunit.client.0.smithi007.stdout:test cmd on pid 40920 2024-01-21T03:21:00.905 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40920 2024-01-21T03:21:00.905 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.906 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.907 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.908 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.909 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s handler_error 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' handler_error' 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.910 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.911 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40782/exe 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.912 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.913 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo handler_error 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40846/exe 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.914 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.915 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.915 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.915 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.915 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.915 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2024-01-21T03:21:00.915 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.915 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ list_parallel: /' 2024-01-21T03:21:00.915 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=handler_error 2024-01-21T03:21:00.915 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40968 2024-01-21T03:21:00.915 INFO:tasks.workunit.client.0.smithi007.stdout:test handler_error on pid 40968 2024-01-21T03:21:00.921 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test handler_error on pid 40968' 2024-01-21T03:21:00.921 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40968 2024-01-21T03:21:00.921 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T03:21:00.921 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.921 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.921 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.921 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.921 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.921 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.921 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s io 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' io' 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.922 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo io 2024-01-21T03:21:00.923 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.925 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.925 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.925 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.925 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_list_parallel.log 2024-01-21T03:21:00.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.925 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=io 2024-01-21T03:21:00.925 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40987 2024-01-21T03:21:00.925 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test io on pid 40987' 2024-01-21T03:21:00.925 INFO:tasks.workunit.client.0.smithi007.stdout:test io on pid 40987 2024-01-21T03:21:00.926 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40987 2024-01-21T03:21:00.926 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T03:21:00.926 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.926 INFO:tasks.workunit.client.0.smithi007.stdout:0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.554+0000 7f588381c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.554+0000 7f588381c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.554+0000 7f588381c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.564+0000 7f588301b640 10 monclient(hunting): get_auth_request con 0x558e7ef6dae0 auth_method 0 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.564+0000 7f588301b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.564+0000 7f588301b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.564+0000 7f588301b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.564+0000 7f588281a640 10 monclient(hunting): get_auth_request con 0x558e7efa5c30 auth_method 0 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.564+0000 7f588281a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.564+0000 7f588281a640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.564+0000 7f588281a640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.581+0000 7f588381c640 10 monclient(hunting): handle_auth_done global_id 4386 payload 824 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.581+0000 7f588381c640 1 -- 172.21.15.7:0/2058430812 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x558e7efa5c30 msgr2=0x558e7f034ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:21:00.927 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.581+0000 7f588381c640 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x558e7efa5c30 0x558e7f034ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.581+0000 7f588381c640 1 -- 172.21.15.7:0/2058430812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e7ef6dae0 msgr2=0x558e7f042d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.581+0000 7f588381c640 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e7ef6dae0 0x558e7f042d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.581+0000 7f588381c640 10 monclient: _finish_hunting 0 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.581+0000 7f588381c640 1 monclient: found mon.noname-a 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.581+0000 7f588381c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.581+0000 7f588381c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.581+0000 7f588381c640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5874005ca0 con 0x558e7f03d5c0 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.650+0000 7f5880816640 10 client.?.objecter ms_handle_connect 0x558e7f03d5c0 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.650+0000 7f5880816640 10 client.?.objecter resend_mon_ops 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.650+0000 7f588381c640 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558e7f03d5c0 0x558e7f0dbd10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f587c00f870 tx=0x7f587c00fd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.758+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c002470 con 0x558e7f03d5c0 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.758+0000 7f5880816640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.758+0000 7f5880816640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.758+0000 7f5880816640 10 monclient: dump: 2024-01-21T03:21:00.928 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 1 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 19 (squid) 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.758+0000 7f5880816640 10 monclient: _finish_auth 0 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.758+0000 7f5880816640 10 monclient: _check_auth_tickets 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.758+0000 7f5880816640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.758+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587c002e30 con 0x558e7f03d5c0 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.758+0000 7f5880816640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:21:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c00db60 con 0x558e7f03d5c0 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5880816640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5880816640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5880816640 10 monclient: dump: 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 1 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 19 (squid) 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5885c4f780 5 monclient: authenticate success, global_id 4386 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5885c4f780 10 monclient: _renew_subs 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5885c4f780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:21:00.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558e7eecbba0 con 0x558e7f03d5c0 2024-01-21T03:21:00.931 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5885c4f780 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:21:00.931 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5885c4f780 10 monclient: _renew_subs 2024-01-21T03:21:00.931 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5885c4f780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:21:00.931 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e7eebed50 con 0x558e7f03d5c0 2024-01-21T03:21:00.935 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s ec_io 2024-01-21T03:21:00.935 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' ec_io' 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo ec_io 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.936 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.937 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.938 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.938 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=ec_io 2024-01-21T03:21:00.938 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41006 2024-01-21T03:21:00.938 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test ec_io on pid 41006' 2024-01-21T03:21:00.938 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 202test ec_io on pid 41006 2024-01-21T03:21:00.943 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=41006 2024-01-21T03:21:00.943 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T03:21:00.943 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.943 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-01-21T03:21:00.943 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.943 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-01-21T03:21:00.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s list 2024-01-21T03:21:00.943 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' list' 2024-01-21T03:21:00.943 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo list 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.947 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.947 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=list 2024-01-21T03:21:00.947 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41029 2024-01-21T03:21:00.947 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test list on pid 41029' 2024-01-21T03:21:00.947 INFO:tasks.workunit.client.0.smithi007.stdout:test list on pid 41029 2024-01-21T03:21:00.968 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=41029 2024-01-21T03:21:00.968 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.969 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.970 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.971 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.972 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40883/exe 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_neorados_cls 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_neorados_cls.log 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ cls: /' 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40920/exe 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_neorados_cmd 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s ec_list 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' ec_list' 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:00.981 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:00.982 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:00.983 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo ec_list 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:00.984 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_neorados_cmd.log 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ cmd: /' 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:00.985 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:00.986 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:00.986 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.986 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:00.986 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.986 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:00.986 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:00.986 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.986 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:00.986 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.986 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:00.986 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:00.986 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=ec_list 2024-01-21T03:21:00.986 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41124 2024-01-21T03:21:00.987 INFO:tasks.workunit.client.0.smithi007.stdout:test ec_list on pid 41124 2024-01-21T03:21:00.991 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test ec_list on pid 41124' 2024-01-21T03:21:00.991 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=41124 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s misc 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' misc' 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/41006/exe 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:00.992 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:00.993 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:00.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:00.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:00.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:00.995 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:00.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:00.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:00.995 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:00.995 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:00.995 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:00.995 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:00.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo misc 2024-01-21T03:21:00.995 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=misc 2024-01-21T03:21:00.995 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41152 2024-01-21T03:21:00.996 INFO:tasks.workunit.client.0.smithi007.stdout:test misc on pid 41152 2024-01-21T03:21:01.010 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test misc on pid 41152' 2024-01-21T03:21:01.010 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=41152 2024-01-21T03:21:01.010 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T03:21:01.010 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:01.010 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:01.010 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:01.010 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_neorados_ec_io 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:01.011 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.012 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.014 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:01.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s pool 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.016 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' pool' 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_neorados_ec_io.log 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ ec_io: /' 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40987/exe 2024-01-21T03:21:01.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40968/exe 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:01.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo pool 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:01.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_neorados_io 2024-01-21T03:21:01.021 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:01.022 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=pool 2024-01-21T03:21:01.022 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_neorados_io.log 2024-01-21T03:21:01.022 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:01.022 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41204 2024-01-21T03:21:01.022 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:01.022 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:01.022 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:01.022 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-01-21T03:21:01.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:01.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:01.022 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test pool on pid 41204' 2024-01-21T03:21:01.022 INFO:tasks.workunit.client.0.smithi007.stdout:test pool on pid 41204 2024-01-21T03:21:01.032 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=41204 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_neorados_handler_error.log 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ io: /' 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.033 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ handler_error: /' 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:01.034 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_neorados_handler_error 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:01.035 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s read_operations 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' read_operations' 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:01.038 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:01.039 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo read_operations 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:01.040 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.041 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/41029/exe 2024-01-21T03:21:01.041 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:01.041 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:01.041 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:01.041 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.041 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:01.041 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.041 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:01.041 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:01.041 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.041 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:01.041 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.041 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:01.042 INFO:tasks.workunit.client.0.smithi007.stdout:test read_operations on pid 41234 2024-01-21T03:21:01.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:01.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:01.054 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:01.054 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:01.054 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=read_operations 2024-01-21T03:21:01.054 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41234 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test read_operations on pid 41234' 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=41234 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:01.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:01.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ list: /' 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.057 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_neorados_list.log 2024-01-21T03:21:01.058 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:01.059 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.060 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s snapshots 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' snapshots' 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_neorados_list 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.061 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:01.062 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.063 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/41124/exe 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:01.064 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo snapshots 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:01.065 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:01.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=snapshots 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_neorados_ec_list 2024-01-21T03:21:01.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:01.069 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41290 2024-01-21T03:21:01.069 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:01.069 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test snapshots on pid 41290' 2024-01-21T03:21:01.069 INFO:tasks.workunit.client.0.smithi007.stdout:test snapshots on pid 41290 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=41290 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_neorados_ec_list.log 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ ec_list: /' 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s watch_notify 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' watch_notify' 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-01-21T03:21:01.073 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/41152/exe 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo watch_notify 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:01.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=watch_notify 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41311 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test watch_notify on pid 41311' 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.075 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:01.076 INFO:tasks.workunit.client.0.smithi007.stdout:test watch_notify on pid 41311 2024-01-21T03:21:01.082 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=41311 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s write_operations 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' write_operations' 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:01.083 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/41204/exe 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo write_operations 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:01.084 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ misc: /' 2024-01-21T03:21:01.085 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_neorados_misc 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:01.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:01.087 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_neorados_pool 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_neorados_misc.log 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_neorados_pool.log 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ pool: /' 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=write_operations 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41331 2024-01-21T03:21:01.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test write_operations on pid 41331' 2024-01-21T03:21:01.089 INFO:tasks.workunit.client.0.smithi007.stdout:test write_operations on pid 41331 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=41331 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:+ ret=0 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40263 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40263 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:01.113 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:01.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T03:21:01.116 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/41234/exe 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:01.117 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:01.118 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:01.119 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_neorados_read_operations 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_neorados_read_operations.log 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ read_operations: /' 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T03:21:01.120 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:01.122 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.123 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:01.124 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:01.125 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:01.126 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/41290/exe 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.127 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.128 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.129 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_neorados_snapshots.log 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ snapshots: /' 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_neorados_snapshots 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/41331/exe 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T03:21:01.130 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T03:21:01.131 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/41311/exe 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_neorados_write_operations 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_neorados_write_operations.log 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-01-21T03:21:01.132 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ write_operations: /' 2024-01-21T03:21:01.133 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-01-21T03:21:01.133 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-01-21T03:21:01.133 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-01-21T03:21:01.133 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-01-21T03:21:01.133 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-01-21T03:21:01.133 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-01-21T03:21:01.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T03:21:01.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-01-21T03:21:01.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T03:21:01.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-01-21T03:21:01.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-01-21T03:21:01.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ watch_notify: /' 2024-01-21T03:21:01.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_neorados_watch_notify 2024-01-21T03:21:01.134 INFO:tasks.workunit.client.0.smithi007.stdout:4-01-21T03:21:00.759+0000 7f5885c4f780 1 librados: init done 2024-01-21T03:21:01.134 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5885c4f780 10 librados: wait_for_osdmap waiting 2024-01-21T03:21:01.134 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.759+0000 7f5881818640 4 set_mon_vals no callback set 2024-01-21T03:21:01.134 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.912+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f587c003040 con 0x558e7f03d5c0 2024-01-21T03:21:01.134 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.912+0000 7f5880816640 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f587006aad0 0x7f587006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:21:01.134 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.913+0000 7f588301b640 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f587006aad0 0x7f587006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:21:01.134 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.913+0000 7f588301b640 10 monclient: get_auth_request con 0x7f587006aad0 auth_method 0 2024-01-21T03:21:01.134 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.913+0000 7f588301b640 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f587006aad0 0x7f587006cf90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5874000c00 tx=0x7f5874017040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:21:01.134 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:00.913+0000 7f5880816640 10 client.4386.objecter ms_handle_connect 0x7f587006aad0 2024-01-21T03:21:01.134 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.025+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f587c025300 con 0x558e7f03d5c0 2024-01-21T03:21:01.134 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.025+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(26..26 src has 1..26) v4 2024-01-21T03:21:01.134 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.025+0000 7f5880816640 3 client.4386.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T03:21:01.135 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.025+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding full epoch 26 2024-01-21T03:21:01.135 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.025+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:01.135 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.025+0000 7f5885c4f780 10 librados: wait_for_osdmap done waiting 2024-01-21T03:21:01.135 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.025+0000 7f5885c4f780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:21:01.135 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.025+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x558e7ef1f800 con 0x558e7f03d5c0 2024-01-21T03:21:01.135 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.115+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f587c0573c0 con 0x558e7f03d5c0 2024-01-21T03:21:01.135 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.115+0000 7f5880816640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-21T03:21:01.135 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.115+0000 7f5881017640 10 client.4386.objecter _get_latest_version latest 26, have it 2024-01-21T03:21:01.135 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.116+0000 7f5885c4f780 10 client.4386.objecter create_pool name=ceph_test_rados_api_asio 2024-01-21T03:21:01.135 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.116+0000 7f5885c4f780 10 client.4386.objecter pool_op_submit 1 2024-01-21T03:21:01.135 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.116+0000 7f5885c4f780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:21:03.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:01.116+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x558e7ef30a00 con 0x558e7f03d5c0 2024-01-21T03:21:03.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:02.907+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v28) v1 ==== 43+0+0 (secure 0 0 0) 0x7f587c019050 con 0x558e7f03d5c0 2024-01-21T03:21:03.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:02.907+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 pool_op_reply(tid 1 (0) Success v28) v1 2024-01-21T03:21:03.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:02.907+0000 7f5880816640 10 client.4386.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v28) v1 2024-01-21T03:21:03.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:02.907+0000 7f5880816640 10 client.4386.objecter have request 1 at 0x558e7f0379d0 Op: create 2024-01-21T03:21:03.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:02.907+0000 7f5880816640 20 client.4386.objecter waiting for client to reach epoch 28 before calling back 2024-01-21T03:21:03.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:02.907+0000 7f5880816640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:21:03.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:02.907+0000 7f5880816640 10 monclient: _renew_subs 2024-01-21T03:21:03.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:02.907+0000 7f5880816640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:02.907+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=27}) v3 -- 0x7f5870070810 con 0x558e7f03d5c0 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:02.907+0000 7f5880816640 10 client.4386.objecter done 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 8 ==== osd_map(27..28 src has 1..28) v4 ==== 14938+0+0 (secure 0 0 0) 0x7f587c057cf0 con 0x558e7f03d5c0 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(27..28 src has 1..28) v4 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5880816640 3 client.4386.objecter handle_osd_map got epochs [27,28] > 26 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 27 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 28 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 10 librados: set snap read head -> 1 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 10 librados: writefull oid=exist nspace= 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f0dc250 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 28 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x558e7f037060 0x558e7f035630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:21:03.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:03.252 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [writefull 0~5 in=5b] tid 2 osd.7 2024-01-21T03:21:03.252 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:09.519 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 2 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 15 client.4386.objecter _send_op 2 to 14.1 on osd.7 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) v8 -- 0x558e7f0e8060 con 0x558e7f037060 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.264+0000 7f5885c4f780 5 client.4386.objecter 1 in flight 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.278+0000 7f588281a640 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x558e7f037060 0x558e7f035630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.279+0000 7f588281a640 10 monclient: get_auth_request con 0x558e7f037060 auth_method 0 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.294+0000 7f588281a640 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x558e7f037060 0x558e7f035630 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.294+0000 7f5880816640 10 client.4386.objecter ms_handle_connect 0x558e7f037060 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.533+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.533+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.533+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:03.533+0000 7f586b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:04.533+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:09.520 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:04.533+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:04.533+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:05.533+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:05.533+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:05.533+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:05.760+0000 7f588527f640 10 client.4386.objecter tick 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:06.534+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:06.534+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:06.534+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:07.534+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:07.534+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:07.534+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:08.534+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:08.534+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:08.534+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:09.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:09.534+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:09.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:09.534+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:12.047 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T0 api_service: [==========] Running 4 tests from 1 test suite. 2024-01-21T03:21:12.047 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [----------] Global test environment set-up. 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [----------] 4 tests from LibRadosService 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5288 ms) 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ OK ] LibRadosService.RegisterLate (809 ms) 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: cluster: 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: id: 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: health: HEALTH_WARN 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 22 pool(s) do not have an application enabled 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: services: 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-01-21T03:21:12.048 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: osd: 8 osds: 8 up (since 71s), 8 in (since 2m) 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: data: 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pools: 35 pools, 1065 pgs 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: objects: 2 objects, 449 KiB 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pgs: 93.615% pgs unknown 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 5.540% pgs not active 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 997 unknown 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 59 creating+peering 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 9 active+clean 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-01-21T03:21:12.049 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: cluster: 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: id: 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: health: HEALTH_WARN 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 18 pool(s) do not have an application enabled 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: services: 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: osd: 8 osds: 8 up (since 73s), 8 in (since 2m) 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: foo: 12 portals active (1 hosts, 3 zones) 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: data: 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pools: 38 pools, 1089 pgs 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: objects: 2 objects, 449 KiB 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-01-21T03:21:12.050 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pgs: 66.758% pgs unknown 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 31.956% pgs not active 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 727 unknown 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 337 creating+peering 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 14 active+clean 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 11 creating+activating 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: cluster: 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: id: 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: health: HEALTH_WARN 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 22 pool(s) do not have an application enabled 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: services: 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-01-21T03:21:12.051 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: osd: 8 osds: 8 up (since 75s), 8 in (since 2m) 2024-01-21T03:21:12.052 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-01-21T03:21:12.052 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-01-21T03:21:12.052 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-01-21T03:21:12.052 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: data: 2024-01-21T03:21:12.052 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pools: 41 pools, 1209 pgs 2024-01-21T03:21:12.052 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: objects: 2 objects, 449 KiB 2024-01-21T03:21:15.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: usage: 214 MiB used, 720 3:21:09.534+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:15.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:10.534+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:15.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:10.534+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:15.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:10.534+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:15.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:10.761+0000 7f588527f640 10 client.4386.objecter tick 2024-01-21T03:21:15.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:11.534+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:11.534+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:11.534+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:12.534+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:12.534+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:12.534+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:13.534+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:13.534+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:13.534+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:13.534+0000 7f586b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:14.534+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:14.534+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:14.534+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.535+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:15.974 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.535+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:15.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.535+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:15.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.762+0000 7f588527f640 10 client.4386.objecter tick 2024-01-21T03:21:15.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.762+0000 7f588527f640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-01-21T03:21:15.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.762+0000 7f588527f640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:21:15.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.762+0000 7f588527f640 10 monclient: _renew_subs 2024-01-21T03:21:15.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.762+0000 7f588527f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:21:15.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.762+0000 7f588527f640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=29}) v3 -- 0x7f5854002f90 con 0x558e7f03d5c0 2024-01-21T03:21:15.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.762+0000 7f588527f640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- ping magic: 0 v1 -- 0x7f5854003320 con 0x558e7f037060 2024-01-21T03:21:15.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 9 ==== osd_map(29..36 src has 1..36) v4 ==== 20508+0+0 (secure 0 0 0) 0x7f587c05b740 con 0x558e7f03d5c0 2024-01-21T03:21:15.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(29..36 src has 1..36) v4 2024-01-21T03:21:15.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 3 client.4386.objecter handle_osd_map got epochs [29,36] > 28 2024-01-21T03:21:15.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 29 2024-01-21T03:21:15.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 29 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 30 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 30 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 31 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 31 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 32 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 32 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:15.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 33 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 33 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 34 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 34 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.987+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 35 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.988+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.988+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 35 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.988+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.988+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 36 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.988+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:15.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.988+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 36 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:17.289 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:31.618 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:17.292 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:31.622 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:17.601 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:31.930 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:17.664 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:31.993 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:17.786 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:32.116 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:17.874 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:32.204 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:17.876 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:32.151 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:17.900 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:32.175 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:18.051 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:32.326 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:20.979 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.988+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:20.979 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.988+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:15.988+0000 7f5880816640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:16.535+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:16.535+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:16.535+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:17.535+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:17.535+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:17.535+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:18.535+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:18.535+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:18.535+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:19.535+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:19.535+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:19.535+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:20.980 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.535+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.535+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.535+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.763+0000 7f588527f640 10 client.4386.objecter tick 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.763+0000 7f588527f640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.763+0000 7f588527f640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.763+0000 7f588527f640 10 monclient: _renew_subs 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.763+0000 7f588527f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.763+0000 7f588527f640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=37}) v3 -- 0x7f58540034e0 con 0x558e7f03d5c0 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.763+0000 7f588527f640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- ping magic: 0 v1 -- 0x7f58540036d0 con 0x558e7f037060 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 10 ==== osd_map(37..39 src has 1..39) v4 ==== 2302+0+0 (secure 0 0 0) 0x7f587c055730 con 0x558e7f03d5c0 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(37..39 src has 1..39) v4 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 3 client.4386.objecter handle_osd_map got epochs [37,39] > 36 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 37 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 37 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:20.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:25.358 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 pool: Running main() from gmock_main.cc 2024-01-21T03:21:25.358 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-01-21T03:21:25.358 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [----------] Global test environment set-up. 2024-01-21T03:21:25.358 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-01-21T03:21:25.358 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-01-21T03:21:25.358 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [ OK ] NeoRadosPools.PoolList (4197 ms) 2024-01-21T03:21:25.358 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-01-21T03:21:25.358 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (3168 ms) 2024-01-21T03:21:25.358 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-01-21T03:21:25.358 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (3369 ms) 2024-01-21T03:21:25.358 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-01-21T03:21:25.359 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (8406 ms) 2024-01-21T03:21:25.359 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-01-21T03:21:25.359 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2733 ms) 2024-01-21T03:21:25.359 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-01-21T03:21:25.359 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2586 ms) 2024-01-21T03:21:25.359 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [----------] 6 tests from NeoRadosPools (24459 ms total) 2024-01-21T03:21:25.359 INFO:tasks.workunit.client.0.smithi007.stdout: pool: 2024-01-21T03:21:25.359 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [----------] Global test environment tear-down 2024-01-21T03:21:25.359 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [==========] 6 tests from 1 test suite ran. (24459 ms total) 2024-01-21T03:21:25.359 INFO:tasks.workunit.client.0.smithi007.stdout: pool: [ PASSED ] 6 tests. 2024-01-21T03:21:25.762 INFO:tasks.workunit.client.0.smithi007.stdout: 3 client.4386.objecter handle_osd_map decoding incremental epoch 38 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 38 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 39 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 39 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:20.994+0000 7f5880816640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:21.535+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:21.535+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:21.535+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:22.535+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:25.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:22.535+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:22.535+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:23.535+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:23.535+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:23.535+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:23.535+0000 7f586b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:24.535+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:24.535+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:24.535+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.535+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.536+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.536+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.764+0000 7f588527f640 10 client.4386.objecter tick 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.764+0000 7f588527f640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.764+0000 7f588527f640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:21:25.764 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.764+0000 7f588527f640 10 monclient: _renew_subs 2024-01-21T03:21:25.765 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.764+0000 7f588527f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:21:25.765 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.764+0000 7f588527f640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=40}) v3 -- 0x7f5854003860 con 0x558e7f03d5c0 2024-01-21T03:21:25.765 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.764+0000 7f588527f640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- ping magic: 0 v1 -- 0x7f5854003b00 con 0x558e7f037060 2024-01-21T03:21:29.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 11 ==== osd_map(40..43 src has 1..43) v4 ==== 3970+0+0 (secure 0 0 0) 0x7f587c057520 con 0x558e7f03d5c0 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(40..43 src has 1..43) v4 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 3 client.4386.objecter handle_osd_map got epochs [40,43] > 39 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 40 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 40 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 41 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 41 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 42 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 42 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:29.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 43 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 43 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:25.777+0000 7f5880816640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:26.536+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:26.536+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:26.536+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:27.536+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:27.536+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:27.536+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:28.536+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:28.536+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:28.536+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:29.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:29.536+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:31.521 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:29.536+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:31.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:29.536+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:31.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.536+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:31.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.536+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:31.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.536+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:31.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.765+0000 7f588527f640 10 client.4386.objecter tick 2024-01-21T03:21:31.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.765+0000 7f588527f640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-01-21T03:21:31.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.765+0000 7f588527f640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:21:31.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.765+0000 7f588527f640 10 monclient: _renew_subs 2024-01-21T03:21:31.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.765+0000 7f588527f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:21:31.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.765+0000 7f588527f640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=44}) v3 -- 0x7f5854003c90 con 0x558e7f03d5c0 2024-01-21T03:21:31.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.765+0000 7f588527f640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- ping magic: 0 v1 -- 0x7f5854003e20 con 0x558e7f037060 2024-01-21T03:21:31.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 12 ==== osd_map(44..46 src has 1..46) v4 ==== 1543+0+0 (secure 0 0 0) 0x7f587c053d00 con 0x558e7f03d5c0 2024-01-21T03:21:31.522 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(44..46 src has 1..46) v4 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 3 client.4386.objecter handle_osd_map got epochs [44,46] > 43 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 44 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 44 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 45 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 45 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 46 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 46 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:30.869+0000 7f5880816640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-01-21T03:21:31.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:31.536+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:32.076 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:2 api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-01-21T03:21:32.076 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [----------] Global test environment set-up. 2024-01-21T03:21:32.076 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-01-21T03:21:32.076 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-01-21T03:21:32.076 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5499 ms) 2024-01-21T03:21:32.076 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (444 ms) 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20133 ms) 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: attempt 0 of 20 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: failed to deregister: 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: { 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "epoch": 12, 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "modified": "2024-01-21T03:21:27.541059+0000", 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "services": { 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "laundry": { 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "daemons": { 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "summary": "", 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "close-test-pid40810": { 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "start_epoch": 12, 2024-01-21T03:21:32.077 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "start_stamp": "2024-01-21T03:21:26.983865+0000", 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "gid": 4607, 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "addr": "172.21.15.7:0/3014332975", 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "metadata": { 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "arch": "x86_64", 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_release": "squid", 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-766-g0edf41a6 (0edf41a622739843d6b978b179ff3227b476dd9d) squid (dev)", 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_version_short": "19.0.0-766-g0edf41a6", 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "distro": "centos", 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "distro_version": "9", 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "foo": "bar", 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "hostname": "smithi007", 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Jan 16 22:14:05 UTC 2024", 2024-01-21T03:21:32.078 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "kernel_version": "5.14.0-409.el9.x86_64", 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "mem_swap_kb": "0", 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "mem_total_kb": "32494756", 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "os": "Linux", 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "this": "that" 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: }, 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "task_status": {} 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: }, 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "pid40771": { 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "start_epoch": 11, 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "start_stamp": "2024-01-21T03:21:12.186830+0000", 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "gid": 4586, 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "addr": "172.21.15.7:0/3728852440", 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "metadata": { 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "arch": "x86_64", 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_release": "squid", 2024-01-21T03:21:32.079 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-766-g0edf41a6 (0edf41a622739843d6b978b179ff3227b476dd9d) squid (dev)", 2024-01-21T03:21:32.080 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "ceph_version_short": "19.0.0-766-g0edf41a6", 2024-01-21T03:21:32.080 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-01-21T03:21:32.159 INFO:tasks.workunit.client.0.smithi007.stdout: GiB / 720 GiB avail 2024-01-21T03:21:32.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pgs: 41.853% pgs unknown 2024-01-21T03:21:32.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 55.335% pgs not active 2024-01-21T03:21:32.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 564 creating+peering 2024-01-21T03:21:32.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 506 unknown 2024-01-21T03:21:32.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 105 creating+activating 2024-01-21T03:21:32.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 34 active+clean 2024-01-21T03:21:32.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-01-21T03:21:32.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-01-21T03:21:32.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5302 ms) 2024-01-21T03:21:32.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ RUN ] LibRadosService.Status 2024-01-21T03:21:32.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ OK ] LibRadosService.Status (20073 ms) 2024-01-21T03:21:32.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [----------] 4 tests from LibRadosService (31472 ms total) 2024-01-21T03:21:32.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-01-21T03:21:32.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [----------] Global test environment tear-down 2024-01-21T03:21:32.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31472 ms total) 2024-01-21T03:21:32.160 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ PASSED ] 4 tests. 2024-01-21T03:21:33.414 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-01-21T03:21:33.414+0000 ea4f640 -1 osd.7 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4386.0:2 14.1 14.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:34.896 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-01-21T03:21:34.733+0000 ea4f640 -1 osd.7 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4386.0:2 14.1 14.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:35.194 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:21:35.071+0000 12a57640 -1 osd.6 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4449.0:2 16.1a 16:5c6776db:::0.obj:head [write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:35.735 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-01-21T03:21:35.730+0000 ea4f640 -1 osd.7 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4386.0:2 14.1 14.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:35.884 INFO:tasks.workunit.client.0.smithi007.stdout:1:31.536+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:35.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:31.536+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:35.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:32.536+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:35.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:32.536+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:35.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:32.536+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:35.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:33.536+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:35.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:33.536+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:35.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:33.536+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:35.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:33.536+0000 7f586b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:21:35.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:34.536+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:34.536+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:34.536+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.537+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.537+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.537+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.766+0000 7f588527f640 10 client.4386.objecter tick 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.766+0000 7f588527f640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.766+0000 7f588527f640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.766+0000 7f588527f640 10 monclient: _renew_subs 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.766+0000 7f588527f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.766+0000 7f588527f640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=47}) v3 -- 0x7f5854003c90 con 0x558e7f03d5c0 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.766+0000 7f588527f640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- ping magic: 0 v1 -- 0x7f5854004010 con 0x558e7f037060 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.898+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 13 ==== osd_map(47..48 src has 1..48) v4 ==== 1311+0+0 (secure 0 0 0) 0x7f587c00dcc0 con 0x558e7f03d5c0 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.898+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(47..48 src has 1..48) v4 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.898+0000 7f5880816640 3 client.4386.objecter handle_osd_map got epochs [47,48] > 46 2024-01-21T03:21:35.885 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.898+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 47 2024-01-21T03:21:35.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.898+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:35.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.898+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 47 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:35.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.898+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:35.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.898+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 48 2024-01-21T03:21:35.886 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.898+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:36.126 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:21:36.070+0000 ea4f640 -1 osd.4 48 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4452.0:2 27.0 27.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:36.531 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:21:36.327+0000 12a57640 -1 osd.3 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4395.0:2 5.7 5:e6ca39de:test-rados-api-smithi007-40276-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:36.901 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-01-21T03:21:36.788+0000 ea4f640 -1 osd.7 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:37.004 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.898+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 48 base exist handler_error: Running main() from gmock_main.cc 2024-01-21T03:21:37.004 INFO:tasks.workunit.client.0.smithi007.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-01-21T03:21:37.004 INFO:tasks.workunit.client.0.smithi007.stdout: handler_error: [----------] Global test environment set-up. 2024-01-21T03:21:37.004 INFO:tasks.workunit.client.0.smithi007.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-01-21T03:21:37.004 INFO:tasks.workunit.client.0.smithi007.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-01-21T03:21:37.004 INFO:tasks.workunit.client.0.smithi007.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (36153 ms) 2024-01-21T03:21:37.004 INFO:tasks.workunit.client.0.smithi007.stdout: handler_error: [----------] 1 test from neocls_handler_error (36153 ms total) 2024-01-21T03:21:37.004 INFO:tasks.workunit.client.0.smithi007.stdout: handler_error: 2024-01-21T03:21:37.005 INFO:tasks.workunit.client.0.smithi007.stdout: handler_error: [----------] Global test environment tear-down 2024-01-21T03:21:37.005 INFO:tasks.workunit.client.0.smithi007.stdout: handler_error: [==========] 1 test from 1 test suite ran. (36154 ms total) 2024-01-21T03:21:37.005 INFO:tasks.workunit.client.0.smithi007.stdout: handler_error: [ PASSED ] 1 test. 2024-01-21T03:21:37.146 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:21:37.107+0000 ea4f640 -1 osd.4 48 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4452.0:2 27.0 27.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:37.386 INFO:tasks.workunit.client.0.smithi007.stdout:api_service_pp: "distro": "centos", 2024-01-21T03:21:37.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "distro_version": "9", 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "foo": "bar", 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "hostname": "smithi007", 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Jan 16 22:14:05 UTC 2024", 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "kernel_version": "5.14.0-409.el9.x86_64", 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "mem_swap_kb": "0", 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "mem_total_kb": "32494756", 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "os": "Linux", 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "this": "that" 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: }, 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: "task_status": {} 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: } 2024-01-21T03:21:37.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: 2024-01-21T03:21:37.388 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: attempt 1 of 20 2024-01-21T03:21:37.388 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10613 ms) 2024-01-21T03:21:37.388 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (36689 ms total) 2024-01-21T03:21:37.388 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: 2024-01-21T03:21:37.388 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [----------] Global test environment tear-down 2024-01-21T03:21:37.388 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (36689 ms total) 2024-01-21T03:21:37.388 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-01-21T03:21:37.534 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:21:37.449+0000 12a57640 -1 osd.3 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4395.0:2 5.7 5:e6ca39de:test-rados-api-smithi007-40276-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:37.973 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-01-21T03:21:37.974+0000 ea4f640 -1 osd.7 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:38.562 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:21:38.427+0000 ea4f640 -1 osd.4 49 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4452.0:2 27.0 27.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:39.216 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-01-21T03:21:39.101+0000 ea4f640 -1 osd.7 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:39.480 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:21:39.470+0000 ea4f640 -1 osd.4 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4452.0:2 27.0 27.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:40.436 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-01-21T03:21:40.348+0000 ea4f640 -1 osd.7 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:40.493 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:21:40.465+0000 ea4f640 -1 osd.4 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4452.0:2 27.0 27.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:40.755 INFO:tasks.workunit.client.0.smithi007.stdout: @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:40.755 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.898+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:40.755 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.898+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:35.898+0000 7f5880816640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:36.537+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:36.537+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:36.537+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:37.537+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:37.537+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:37.537+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:38.473+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 1 ==== osd_map(29..48 src has 1..48) v4 ==== 29490+0+0 (crc 0 0 0) 0x7f58780091f0 con 0x558e7f037060 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:38.473+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(29..48 src has 1..48) v4 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:38.473+0000 7f5880816640 3 client.4386.objecter handle_osd_map ignoring epochs [29,48] <= 48 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:38.473+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:38.473+0000 7f5880816640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:38.537+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:38.537+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:38.537+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:40.756 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:39.537+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:40.757 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:39.537+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:40.757 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:39.537+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:40.757 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.537+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:40.757 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.537+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:40.757 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.537+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:40.757 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.767+0000 7f588527f640 10 client.4386.objecter tick 2024-01-21T03:21:40.757 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.767+0000 7f588527f640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-01-21T03:21:40.757 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.767+0000 7f588527f640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:21:40.757 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.767+0000 7f588527f640 10 monclient: _renew_subs 2024-01-21T03:21:40.757 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.767+0000 7f588527f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:21:40.757 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.767+0000 7f588527f640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=49}) v3 -- 0x7f5854003c90 con 0x558e7f03d5c0 2024-01-21T03:21:40.757 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.767+0000 7f588527f640 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- ping magic: 0 v1 -- 0x7f5854004200 con 0x558e7f037060 2024-01-21T03:21:41.604 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:21:41.612+0000 ea4f640 -1 osd.4 51 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4452.0:2 27.0 27.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:42.277 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== mon.0 v2:172.21.15.7:3300/0 14 ==== osd_map(49..52 src has 1..52) v4 ==== 3648+0+0 (secure 0 0 0) 0x7f587c00de80 con 0x558e7f03d5c0 2024-01-21T03:21:42.278 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(49..52 src has 1..52) v4 2024-01-21T03:21:42.278 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 3 client.4386.objecter handle_osd_map got epochs [49,52] > 48 2024-01-21T03:21:42.278 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 49 2024-01-21T03:21:42.278 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:42.278 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 49 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:42.278 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:42.278 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 50 2024-01-21T03:21:42.278 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:42.278 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 50 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:42.278 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:42.278 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 51 2024-01-21T03:21:42.278 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:42.278 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 51 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 52 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 10 client.4386.objecter checking op 2 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 52 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:40.770+0000 7f5880816640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:41.537+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:41.537+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:41.537+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 2 ==== osd_op_reply(2 exist [writefull 0~5] v48'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(2 exist [writefull 0~5] v48'1 uv1 ondisk = 0) v8 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 2 ondisk uv 1 in 14.1 attempt 0 2024-01-21T03:21:42.279 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 0 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 2 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f588281a640 15 client.4386.objecter _finish_op 2 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 2 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f588281a640 5 client.4386.objecter 0 in flight 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 10 librados: Objecter returned from writefull r=0 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f0dc250 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 52 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [read 0~256] tid 3 osd.7 2024-01-21T03:21:42.280 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 3 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 15 client.4386.objecter _send_op 3 to 14.1 on osd.7 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:3 14.1 14:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52) v8 -- 0x558e7f0ded20 con 0x558e7f037060 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 5 client.4386.objecter 1 in flight 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f0df080 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 52 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.a7bbaa1 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid noexist '@14' '@14' [read 0~256] tid 4 osd.7 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 4 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 15 client.4386.objecter _send_op 4 to 14.1 on osd.7 2024-01-21T03:21:42.281 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:4 14.1 14:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52) v8 -- 0x558e7f0df520 con 0x558e7f037060 2024-01-21T03:21:42.377 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 5 2024-01-21T03:21:42.377 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.292+0000 7f5885c4f780 5 client.4386.objecter 2 in flight 2024-01-21T03:21:42.377 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.386+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:42.377 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.386+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-01-21T03:21:42.377 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.387+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.387+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 3 ondisk uv 1 in 14.1 attempt 0 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.387+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 5 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.387+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 3 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.387+0000 7f588281a640 15 client.4386.objecter _finish_op 3 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.387+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 5 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.387+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 3 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.387+0000 7f588281a640 5 client.4386.objecter 1 in flight 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (100 ms) 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 4 ondisk uv 0 in 14.1 attempt 0 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f588281a640 10 client.4386.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-21T03:21:42.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 0 2024-01-21T03:21:42.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 4 2024-01-21T03:21:42.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f588281a640 15 client.4386.objecter _finish_op 4 2024-01-21T03:21:42.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:42.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 4 2024-01-21T03:21:42.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f588281a640 5 client.4386.objecter 0 in flight 2024-01-21T03:21:42.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f0df080 2024-01-21T03:21:42.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 52 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:42.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:42.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:42.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:42.393 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [read 0~256] tid 5 osd.7 2024-01-21T03:21:42.393 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:42.393 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 5 2024-01-21T03:21:42.393 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 15 client.4386.objecter _send_op 5 to 14.1 on osd.7 2024-01-21T03:21:42.393 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:5 14.1 14:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52) v8 -- 0x558e7f0df8b0 con 0x558e7f037060 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 5 client.4386.objecter 1 in flight 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f0dc250 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 52 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.a7bbaa1 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid noexist '@14' '@14' [read 0~256] tid 6 osd.7 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 6 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 15 client.4386.objecter _send_op 6 to 14.1 on osd.7 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:6 14.1 14:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52) v8 -- 0x558e7f0e83c0 con 0x558e7f037060 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 5 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.392+0000 7f5885c4f780 5 client.4386.objecter 2 in flight 2024-01-21T03:21:42.394 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:42.395 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-01-21T03:21:42.395 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:42.395 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 5 ondisk uv 1 in 14.1 attempt 0 2024-01-21T03:21:42.395 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 5 2024-01-21T03:21:42.395 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 5 2024-01-21T03:21:42.395 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 15 client.4386.objecter _finish_op 5 2024-01-21T03:21:42.395 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 5 2024-01-21T03:21:42.395 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 5 2024-01-21T03:21:42.395 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 5 client.4386.objecter 1 in flight 2024-01-21T03:21:42.395 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:42.395 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-21T03:21:42.395 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:42.396 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 6 ondisk uv 0 in 14.1 attempt 0 2024-01-21T03:21:42.396 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 10 client.4386.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-21T03:21:42.396 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 0 2024-01-21T03:21:42.396 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 6 2024-01-21T03:21:42.396 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 15 client.4386.objecter _finish_op 6 2024-01-21T03:21:42.396 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:42.396 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 6 2024-01-21T03:21:42.396 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.408+0000 7f588281a640 5 client.4386.objecter 0 in flight 2024-01-21T03:21:42.396 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (16 ms) 2024-01-21T03:21:42.396 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-01-21T03:21:42.396 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f0df080 2024-01-21T03:21:42.396 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 52 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:42.396 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:42.397 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:42.397 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:42.397 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [read 0~256] tid 7 osd.7 2024-01-21T03:21:42.397 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:42.397 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 7 2024-01-21T03:21:42.397 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 15 client.4386.objecter _send_op 7 to 14.1 on osd.7 2024-01-21T03:21:42.397 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:7 14.1 14:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52) v8 -- 0x558e7f108730 con 0x558e7f037060 2024-01-21T03:21:42.397 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:42.397 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 5 client.4386.objecter 1 in flight 2024-01-21T03:21:42.397 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f128be0 2024-01-21T03:21:42.409 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 52 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:42.409 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.a7bbaa1 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid noexist '@14' '@14' [read 0~256] tid 8 osd.7 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 8 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 15 client.4386.objecter _send_op 8 to 14.1 on osd.7 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:8 14.1 14:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52) v8 -- 0x558e7f1290a0 con 0x558e7f037060 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 5 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.409+0000 7f5885c4f780 5 client.4386.objecter 2 in flight 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 7 ondisk uv 1 in 14.1 attempt 0 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 5 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 7 2024-01-21T03:21:42.410 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 15 client.4386.objecter _finish_op 7 2024-01-21T03:21:42.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 5 2024-01-21T03:21:42.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 7 2024-01-21T03:21:42.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 5 client.4386.objecter 1 in flight 2024-01-21T03:21:42.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:42.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-21T03:21:42.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:42.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 8 ondisk uv 0 in 14.1 attempt 0 2024-01-21T03:21:42.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 10 client.4386.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-21T03:21:42.411 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 0 2024-01-21T03:21:42.986 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:57.316 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:43.021 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:21:42.966+0000 ea4f640 -1 osd.4 53 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4452.0:2 27.0 27.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:43.052 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:57.382 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:43.132 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:57.407 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:43.134 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:57.409 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:43.191 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:57.521 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:43.193 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:57.523 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:43.266 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:4 api_misc: [==========] Running 12 tests from 4 test suites. 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] Global test environment set-up. 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2024-01-21T03:21:00.801+0000 7f744cc0b780 -1 failed for service _ceph-mon._tcp 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2024-01-21T03:21:00.801+0000 7f744cc0b780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (274 ms) 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-01-21T03:21:43.267 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2011 ms) 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2285 ms total) 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03827230 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a041e0e70 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a042b6160 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a042b3fb0 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a035d2170 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a035ce0d0 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03951ab0 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a039524b0 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03952eb0 2024-01-21T03:21:43.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a039538b0 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a039542b0 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03954da0 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03955890 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03956380 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03956e70 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a042919c0 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a042924b0 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a04292fa0 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a04293a90 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a04294580 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a04295070 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03957900 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03958300 2024-01-21T03:21:43.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03958d00 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03952b40 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03952cb0 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a042936e0 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a04293890 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a042941d0 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a042943a0 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03957590 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03957f50 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a03958190 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a04292de0 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x561a042917c0 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 35 aios 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03827230 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a041e0e70 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a042b6160 2024-01-21T03:21:43.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a042b3fb0 2024-01-21T03:21:43.271 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a035d2170 2024-01-21T03:21:43.271 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a035ce0d0 2024-01-21T03:21:43.271 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03951ab0 2024-01-21T03:21:43.271 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a039524b0 2024-01-21T03:21:43.271 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03952eb0 2024-01-21T03:21:43.271 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a039538b0 2024-01-21T03:21:43.271 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a039542b0 2024-01-21T03:21:43.271 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03954da0 2024-01-21T03:21:43.271 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03955890 2024-01-21T03:21:43.271 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03956380 2024-01-21T03:21:43.271 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03956e70 2024-01-21T03:21:43.302 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:57.632 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:43.352 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:57.627 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:43.450 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:57.780 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:21:44.228 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:21:44.169+0000 ea4f640 -1 osd.4 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4452.0:2 27.0 27.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:44.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x562.424+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 8 2024-01-21T03:21:44.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 15 client.4386.objecter _finish_op 8 2024-01-21T03:21:44.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:44.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 8 2024-01-21T03:21:44.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f588281a640 5 client.4386.objecter 0 in flight 2024-01-21T03:21:44.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (16 ms) 2024-01-21T03:21:44.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-01-21T03:21:44.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-01-21T03:21:44.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 20 librados: queue_aio_write 0x558e7ef1f520 completion 0x558e7eec6a90 write_seq 1 2024-01-21T03:21:44.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f128be0 2024-01-21T03:21:44.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 52 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:44.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [write 0~5 in=5b] tid 9 osd.7 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 9 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 15 client.4386.objecter _send_op 9 to 14.1 on osd.7 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:9 14.1 14:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52) v8 -- 0x558e7f0df080 con 0x558e7f037060 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 5 client.4386.objecter 1 in flight 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.424+0000 7f5885c4f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.537+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.537+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:42.537+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:43.537+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:43.537+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:43.537+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:44.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:43.537+0000 7f586b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:21:44.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.378+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 9 ==== osd_op_reply(9 exist [write 0~5] v52'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:44.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.378+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(9 exist [write 0~5] v52'2 uv2 ondisk = 0) v8 2024-01-21T03:21:44.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.378+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:44.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.378+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 9 ondisk uv 2 in 14.1 attempt 0 2024-01-21T03:21:44.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.378+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 0 2024-01-21T03:21:44.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.378+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 9 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.378+0000 7f588281a640 15 client.4386.objecter _finish_op 9 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.378+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.378+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 9 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.378+0000 7f588281a640 5 client.4386.objecter 0 in flight 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.378+0000 7f588281a640 20 librados: complete_aio_write 0x558e7eec6a90 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (1954 ms) 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 20 librados: queue_aio_write 0x558e7ef1f520 completion 0x558e7eeada90 write_seq 2 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f0df080 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 52 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:44.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:44.449 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [write 0~5 in=5b] tid 10 osd.7 2024-01-21T03:21:44.449 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:44.449 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 10 2024-01-21T03:21:44.449 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 15 client.4386.objecter _send_op 10 to 14.1 on osd.7 2024-01-21T03:21:44.449 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:10 14.1 14:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52) v8 -- 0x558e7f0dc250 con 0x558e7f037060 2024-01-21T03:21:44.449 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:44.449 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 5 client.4386.objecter 1 in flight 2024-01-21T03:21:44.449 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.379+0000 7f5885c4f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-01-21T03:21:44.449 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.461+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 10 ==== osd_map(53..54 src has 1..54) v4 ==== 1312+0+0 (crc 0 0 0) 0x7f58780091f0 con 0x558e7f037060 2024-01-21T03:21:44.449 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.462+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(53..54 src has 1..54) v4 2024-01-21T03:21:44.449 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.462+0000 7f5880816640 3 client.4386.objecter handle_osd_map got epochs [53,54] > 52 2024-01-21T03:21:44.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.462+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 53 2024-01-21T03:21:44.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.462+0000 7f5880816640 10 client.4386.objecter checking op 10 2024-01-21T03:21:44.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.462+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 53 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:44.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.462+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:44.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.462+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 54 2024-01-21T03:21:44.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.462+0000 7f5880816640 10 client.4386.objecter checking op 10 2024-01-21T03:21:44.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.462+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 54 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:44.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.462+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:44.732 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.462+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.462+0000 7f5880816640 20 client.4386.objecter 10 14.d3a59201 osd.7 exist [write 0~5 in=5b] 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.538+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.538+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.538+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 11 ==== osd_op_reply(10 exist [write 0~5] v54'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(10 exist [write 0~5] v54'3 uv3 ondisk = 0) v8 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 10 ondisk uv 3 in 14.1 attempt 0 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 0 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 10 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f588281a640 15 client.4386.objecter _finish_op 10 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 10 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f588281a640 5 client.4386.objecter 0 in flight 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f588281a640 20 librados: complete_aio_write 0x558e7eeada90 2024-01-21T03:21:44.733 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (367 ms) 2024-01-21T03:21:44.734 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-01-21T03:21:44.734 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-01-21T03:21:44.734 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 20 librados: queue_aio_write 0x558e7ef1f520 completion 0x558e7f0dc770 write_seq 3 2024-01-21T03:21:44.734 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f10c070 2024-01-21T03:21:44.734 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 54 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:44.734 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:46.374 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:21:46.259+0000 ea4f640 -1 osd.4 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4452.0:2 27.0 27:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:46.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:46.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:46.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [write 0~5 in=5b] tid 11 osd.7 2024-01-21T03:21:46.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:46.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 11 2024-01-21T03:21:46.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 15 client.4386.objecter _send_op 11 to 14.1 on osd.7 2024-01-21T03:21:46.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:11 14.1 14:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54) v8 -- 0x558e7f0df0e0 con 0x558e7f037060 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 5 client.4386.objecter 1 in flight 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:44.746+0000 7f5885c4f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:45.538+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:45.538+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:45.538+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:45.768+0000 7f588527f640 10 client.4386.objecter tick 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.538+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.538+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.538+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 12 ==== osd_op_reply(11 exist [write 0~5] v54'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(11 exist [write 0~5] v54'4 uv4 ondisk = 0) v8 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 11 ondisk uv 4 in 14.1 attempt 0 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 0 2024-01-21T03:21:46.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 11 2024-01-21T03:21:46.723 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f588281a640 15 client.4386.objecter _finish_op 11 2024-01-21T03:21:46.723 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:46.723 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 11 2024-01-21T03:21:46.723 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f588281a640 5 client.4386.objecter 0 in flight 2024-01-21T03:21:46.723 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f588281a640 20 librados: complete_aio_write 0x558e7f0dc770 2024-01-21T03:21:46.723 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1989 ms) 2024-01-21T03:21:46.723 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-01-21T03:21:46.723 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f10ba00 2024-01-21T03:21:47.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 54 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:47.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:47.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:47.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:47.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [read 0~0] tid 12 osd.7 2024-01-21T03:21:47.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:47.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 12 2024-01-21T03:21:47.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 15 client.4386.objecter _send_op 12 to 14.1 on osd.7 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:12 14.1 14:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54) v8 -- 0x558e7f0df0e0 con 0x558e7f037060 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 5 client.4386.objecter 1 in flight 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f10bea0 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 54 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.a7bbaa1 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid noexist '@14' '@14' [read 0~0] tid 13 osd.7 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 13 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 15 client.4386.objecter _send_op 13 to 14.1 on osd.7 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:13 14.1 14:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54) v8 -- 0x558e7f108a90 con 0x558e7f037060 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 5 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:46.735+0000 7f5885c4f780 5 client.4386.objecter 2 in flight 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 13 ==== osd_map(55..56 src has 1..57) v4 ==== 1295+0+0 (crc 0 0 0) 0x7f58780091f0 con 0x558e7f037060 2024-01-21T03:21:47.171 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(55..56 src has 1..57) v4 2024-01-21T03:21:47.172 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 3 client.4386.objecter handle_osd_map got epochs [55,56] > 54 2024-01-21T03:21:47.172 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 55 2024-01-21T03:21:47.181 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 10 client.4386.objecter checking op 12 2024-01-21T03:21:47.181 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 55 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:47.181 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:47.181 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 10 client.4386.objecter checking op 13 2024-01-21T03:21:47.181 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 55 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:47.181 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-01-21T03:21:47.181 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 56 2024-01-21T03:21:47.181 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 10 client.4386.objecter checking op 12 2024-01-21T03:21:47.181 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 56 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:47.181 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:47.181 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 10 client.4386.objecter checking op 13 2024-01-21T03:21:47.181 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 56 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:47.181 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 20 client.4386.objecter 12 14.d3a59201 osd.7 exist [read 0~0] 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.185+0000 7f5880816640 20 client.4386.objecter 13 14.a7bbaa1 osd.7 noexist [read 0~0] 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 14 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 12 ondisk uv 4 in 14.1 attempt 0 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 5 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 12 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 15 client.4386.objecter _finish_op 12 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 5 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 12 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 5 client.4386.objecter 1 in flight 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 15 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:47.182 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 13 ondisk uv 0 in 14.1 attempt 0 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 10 client.4386.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 0 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 13 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 15 client.4386.objecter _finish_op 13 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 13 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f588281a640 5 client.4386.objecter 0 in flight 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (460 ms) 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f10ba00 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 56 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [read 0~0] tid 14 osd.7 2024-01-21T03:21:47.183 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:47.184 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 14 2024-01-21T03:21:47.184 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 15 client.4386.objecter _send_op 14 to 14.1 on osd.7 2024-01-21T03:21:47.184 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:14 14.1 14:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v8 -- 0x558e7f10bea0 con 0x558e7f037060 2024-01-21T03:21:47.184 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:47.184 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 5 client.4386.objecter 1 in flight 2024-01-21T03:21:47.184 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f108a90 2024-01-21T03:21:47.184 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 56 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:47.184 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-01-21T03:21:47.184 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.a7bbaa1 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:47.184 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:47.184 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid noexist '@14' '@14' [read 0~0] tid 15 osd.7 2024-01-21T03:21:47.862 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:21:47.825+0000 ea4f640 -1 osd.4 57 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4452.0:2 27.0 27:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:47.896 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:47.896 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 15 2024-01-21T03:21:47.896 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 15 client.4386.objecter _send_op 15 to 14.1 on osd.7 2024-01-21T03:21:47.896 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:15 14.1 14:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v8 -- 0x558e7f108f30 con 0x558e7f037060 2024-01-21T03:21:47.896 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 5 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.195+0000 7f5885c4f780 5 client.4386.objecter 2 in flight 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.538+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.538+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.538+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 16 ==== osd_map(57..57 src has 1..57) v4 ==== 644+0+0 (crc 0 0 0) 0x7f58780091f0 con 0x558e7f037060 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 17 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(57..57 src has 1..57) v4 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5880816640 3 client.4386.objecter handle_osd_map got epochs [57,57] > 56 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 57 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5880816640 10 client.4386.objecter checking op 14 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 57 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:47.897 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5880816640 10 client.4386.objecter checking op 15 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 57 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5880816640 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5880816640 20 client.4386.objecter 14 14.d3a59201 osd.7 exist [read 0~0] 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5880816640 20 client.4386.objecter 15 14.a7bbaa1 osd.7 noexist [read 0~0] 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 14 ondisk uv 4 in 14.1 attempt 0 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 5 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 14 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 15 client.4386.objecter _finish_op 14 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 5 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 14 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 5 client.4386.objecter 1 in flight 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 18 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:47.898 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 15 ondisk uv 0 in 14.1 attempt 0 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 10 client.4386.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 0 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 15 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 15 client.4386.objecter _finish_op 15 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 15 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f588281a640 5 client.4386.objecter 0 in flight 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (716 ms) 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f108a90 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 57 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:47.899 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:47.900 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [read 0~0] tid 16 osd.7 2024-01-21T03:21:47.900 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:47.900 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 16 2024-01-21T03:21:47.900 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 15 client.4386.objecter _send_op 16 to 14.1 on osd.7 2024-01-21T03:21:47.900 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:16 14.1 14:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57) v8 -- 0x558e7f108f30 con 0x558e7f037060 2024-01-21T03:21:47.900 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:47.900 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 5 client.4386.objecter 1 in flight 2024-01-21T03:21:47.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f109420 2024-01-21T03:21:47.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 57 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T03:21:47.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-01-21T03:21:47.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.a7bbaa1 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:47.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:47.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid noexist '@14' '@14' [read 0~0] tid 17 osd.7 2024-01-21T03:21:47.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:47.929 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 17 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 15 client.4386.objecter _send_op 17 to 14.1 on osd.7 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:17 14.1 14:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57) v8 -- 0x558e7f10c730 con 0x558e7f037060 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 5 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.911+0000 7f5885c4f780 5 client.4386.objecter 2 in flight 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 19 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 16 ondisk uv 4 in 14.1 attempt 0 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 5 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 16 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 15 client.4386.objecter _finish_op 16 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 5 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 16 2024-01-21T03:21:47.930 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 5 client.4386.objecter 1 in flight 2024-01-21T03:21:47.931 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 20 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:47.931 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-21T03:21:47.931 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:47.931 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 17 ondisk uv 0 in 14.1 attempt 0 2024-01-21T03:21:48.369 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:21:48.159+0000 12a57640 -1 osd.0 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:21:48.794 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:21:48.804+0000 ea4f640 -1 osd.4 57 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4452.0:2 27.0 27:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:49.808 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:21:49.813+0000 ea4f640 -1 osd.4 58 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4452.0:2 27.0 27:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:50.273 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 10 client.4386.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-21T03:21:50.273 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 0 2024-01-21T03:21:50.273 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 17 2024-01-21T03:21:50.273 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 15 client.4386.objecter _finish_op 17 2024-01-21T03:21:50.273 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 17 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (33 ms) 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f588281a640 5 client.4386.objecter 0 in flight 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f5885c4f780 20 librados: queue_aio_write 0x558e7ef1f520 completion 0x558e7f10c340 write_seq 4 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f10ba00 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 57 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f5885c4f780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [writefull 0~5 in=5b] tid 18 osd.7 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 18 2024-01-21T03:21:50.274 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f5885c4f780 15 client.4386.objecter _send_op 18 to 14.1 on osd.7 2024-01-21T03:21:50.275 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:18 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57) v8 -- 0x558e7f109420 con 0x558e7f037060 2024-01-21T03:21:50.275 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:50.275 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:47.944+0000 7f5885c4f780 5 client.4386.objecter 1 in flight 2024-01-21T03:21:50.275 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:48.538+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:50.275 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:48.538+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:50.275 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:48.538+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:50.275 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:49.538+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:50.275 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:49.538+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:50.275 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:49.538+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:50.275 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 21 ==== osd_op_reply(18 exist [writefull 0~5] v57'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:50.275 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(18 exist [writefull 0~5] v57'5 uv5 ondisk = 0) v8 2024-01-21T03:21:50.275 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:50.546 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:21:50.520+0000 12a57640 -1 osd.0 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:21:50.561 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 18 ondisk uv 5 in 14.1 attempt 0 2024-01-21T03:21:50.561 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 0 2024-01-21T03:21:50.561 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 18 2024-01-21T03:21:50.561 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f588281a640 15 client.4386.objecter _finish_op 18 2024-01-21T03:21:50.561 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:50.561 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 18 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f588281a640 5 client.4386.objecter 0 in flight 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f588281a640 20 librados: complete_aio_write 0x558e7f10c340 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2344 ms) 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f5885c4f780 20 librados: queue_aio_write 0x558e7ef1f520 completion 0x558e7f0df170 write_seq 5 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f10ba00 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 57 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f5885c4f780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [writefull 0~5 in=5b] tid 19 osd.7 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 19 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f5885c4f780 15 client.4386.objecter _send_op 19 to 14.1 on osd.7 2024-01-21T03:21:50.562 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:19 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57) v8 -- 0x558e7f109420 con 0x558e7f037060 2024-01-21T03:21:50.563 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:50.563 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.288+0000 7f5885c4f780 5 client.4386.objecter 1 in flight 2024-01-21T03:21:50.563 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.538+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:50.563 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.538+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:50.563 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.538+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:50.563 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.576+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 22 ==== osd_map(58..59 src has 1..60) v4 ==== 959+0+0 (crc 0 0 0) 0x7f58780091f0 con 0x558e7f037060 2024-01-21T03:21:50.563 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.576+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(58..59 src has 1..60) v4 2024-01-21T03:21:50.563 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.576+0000 7f5880816640 3 client.4386.objecter handle_osd_map got epochs [58,59] > 57 2024-01-21T03:21:50.904 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:21:50.815+0000 ea4f640 -1 osd.4 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4452.0:2 27.0 27:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T03:21:51.325 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.576+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 58 2024-01-21T03:21:51.325 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.576+0000 7f5880816640 10 client.4386.objecter checking op 19 2024-01-21T03:21:51.325 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.576+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 58 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.576+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.576+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 59 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.576+0000 7f5880816640 10 client.4386.objecter checking op 19 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.576+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 59 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.576+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.576+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.576+0000 7f5880816640 20 client.4386.objecter 19 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:50.769+0000 7f588527f640 10 client.4386.objecter tick 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 23 ==== osd_op_reply(19 exist [writefull 0~5] v59'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(19 exist [writefull 0~5] v59'6 uv6 ondisk = 0) v8 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 19 ondisk uv 6 in 14.1 attempt 0 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 0 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 19 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f588281a640 15 client.4386.objecter _finish_op 19 2024-01-21T03:21:51.326 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:51.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 19 2024-01-21T03:21:51.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f588281a640 5 client.4386.objecter 0 in flight 2024-01-21T03:21:51.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f588281a640 20 librados: complete_aio_write 0x558e7f0df170 2024-01-21T03:21:51.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1052 ms) 2024-01-21T03:21:51.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-01-21T03:21:51.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f5885c4f780 20 librados: queue_aio_write 0x558e7ef1f520 completion 0x558e7f109420 write_seq 6 2024-01-21T03:21:51.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f5885c4f780 10 client.4386.objecter _op_submit op 0x558e7f108a90 2024-01-21T03:21:51.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f5885c4f780 20 client.4386.objecter _calc_target epoch 59 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:51.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f5885c4f780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:51.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f5885c4f780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-01-21T03:21:51.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f5885c4f780 20 client.4386.objecter _get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:51.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f5885c4f780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [writefull 0~5 in=5b] tid 20 osd.7 2024-01-21T03:21:51.565 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:21:51.562+0000 12a57640 -1 osd.0 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:21:52.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f5885c4f780 20 client.4386.objecter get_session s=0x558e7eee3570 osd=7 3 2024-01-21T03:21:52.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f5885c4f780 15 client.4386.objecter _session_op_assign 7 20 2024-01-21T03:21:52.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f5885c4f780 15 client.4386.objecter _send_op 20 to 14.1 on osd.7 2024-01-21T03:21:52.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- osd_op(unknown.0.0:20 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59) v8 -- 0x558e7f108f30 con 0x558e7f037060 2024-01-21T03:21:52.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:52.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.340+0000 7f5885c4f780 5 client.4386.objecter 1 in flight 2024-01-21T03:21:52.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.538+0000 7f586b7fe640 10 monclient: tick 2024-01-21T03:21:52.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.539+0000 7f586b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:21:52.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.539+0000 7f586b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:21:52.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.576+0000 7f5880816640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 24 ==== osd_map(60..60 src has 1..60) v4 ==== 312+0+0 (crc 0 0 0) 0x7f58780091f0 con 0x558e7f037060 2024-01-21T03:21:52.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.576+0000 7f5880816640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_map(60..60 src has 1..60) v4 2024-01-21T03:21:52.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.576+0000 7f5880816640 3 client.4386.objecter handle_osd_map got epochs [60,60] > 59 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.576+0000 7f5880816640 3 client.4386.objecter handle_osd_map decoding incremental epoch 60 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.576+0000 7f5880816640 10 client.4386.objecter checking op 20 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.576+0000 7f5880816640 20 client.4386.objecter _calc_target epoch 60 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.576+0000 7f5880816640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.576+0000 7f5880816640 20 client.4386.objecter dump_active .. 0 homeless 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:51.576+0000 7f5880816640 20 client.4386.objecter 20 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f588281a640 1 -- 172.21.15.7:0/2058430812 <== osd.7 v2:172.21.15.177:6823/4069998673 25 ==== osd_op_reply(20 exist [writefull 0~5] v60'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f587805f410 con 0x558e7f037060 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f588281a640 10 client.4386.objecter ms_dispatch 0x558e7ef18b50 osd_op_reply(20 exist [writefull 0~5] v60'7 uv7 ondisk = 0) v8 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f588281a640 10 client.4386.objecter in handle_osd_op_reply 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f588281a640 7 client.4386.objecter handle_osd_op_reply 20 ondisk uv 7 in 14.1 attempt 0 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f588281a640 10 client.4386.objecter op 0 rval 0 len 0 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f588281a640 15 client.4386.objecter handle_osd_op_reply completed tid 20 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f588281a640 15 client.4386.objecter _finish_op 20 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f588281a640 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 4 2024-01-21T03:21:52.291 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f588281a640 15 client.4386.objecter _session_op_remove 7 20 2024-01-21T03:21:52.292 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f588281a640 5 client.4386.objecter 0 in flight 2024-01-21T03:21:52.294 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f588281a640 20 librados: complete_aio_write 0x558e7f109420 2024-01-21T03:21:52.294 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (964 ms) 2024-01-21T03:21:52.294 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f5885c4f780 10 librados: watch_flush enter 2024-01-21T03:21:52.294 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f5885c4f780 10 librados: watch_flush exit 2024-01-21T03:21:52.294 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f5885c4f780 10 client.4386.objecter close_session for osd.7 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x558e7f037060 msgr2=0x558e7f035630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f5885c4f780 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x558e7f037060 0x558e7f035630 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f5885c4f780 20 client.4386.objecter put_session s=0x558e7eee3570 osd=7 1 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f5885c4f780 20 client.4386.objecter shutdown clearing up homeless session... 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.304+0000 7f5885c4f780 10 client.4386.objecter successfully canceled tick 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.305+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f587006aad0 msgr2=0x7f587006cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.305+0000 7f5885c4f780 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f587006aad0 0x7f587006cf90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5874000c00 tx=0x7f5874017040 comp rx=0 tx=0).stop 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.305+0000 7f5885c4f780 10 monclient: shutdown 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.305+0000 7f5885c4f780 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.305+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558e7f03d5c0 msgr2=0x558e7f0dbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.305+0000 7f5885c4f780 1 --2- 172.21.15.7:0/2058430812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558e7f03d5c0 0x558e7f0dbd10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f587c00f870 tx=0x7f587c00fd30 comp rx=0 tx=0).stop 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.305+0000 7f588381c640 1 -- 172.21.15.7:0/2058430812 reap_dead start 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.305+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 shutdown_connections 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.305+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 >> 172.21.15.7:0/2058430812 conn(0x558e7ef6a520 msgr2=0x558e7f03c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:21:52.295 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.305+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 shutdown_connections 2024-01-21T03:21:52.296 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.305+0000 7f5885c4f780 1 -- 172.21.15.7:0/2058430812 wait complete. 2024-01-21T03:21:52.296 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.305+0000 7f5885c4f780 1 librados: shutdown 2024-01-21T03:21:52.296 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [----------] 12 tests from AsioRados (10012 ms total) 2024-01-21T03:21:52.296 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-01-21T03:21:52.296 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [----------] Global test environment tear-down 2024-01-21T03:21:52.296 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (51811 ms total) 2024-01-21T03:21:52.296 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ PASSED ] 12 tests. 2024-01-21T03:21:55.636 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:21:55.632+0000 12a57640 -1 osd.2 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4401.0:4 13.1b 13:dd8ca106:test-rados-api-smithi007-40363-3::foo:head [call lock.lock in=60b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) 2024-01-21T03:21:56.954 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:21:56.870+0000 12a57640 -1 osd.2 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4401.0:4 13.1b 13:dd8ca106:test-rados-api-smithi007-40363-3::foo:head [call lock.lock in=60b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) 2024-01-21T03:21:57.901 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:21:57.803+0000 12a57640 -1 osd.0 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:21:58.039 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:21:57.971+0000 12a57640 -1 osd.2 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4401.0:4 13.1b 13:dd8ca106:test-rados-api-smithi007-40363-3::foo:head [call lock.lock in=60b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) 2024-01-21T03:21:59.065 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:21:59.058+0000 12a57640 -1 osd.0 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:21:59.656 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:21:59.566+0000 12a57640 -1 osd.2 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4401.0:4 13.1b 13:dd8ca106:test-rados-api-smithi007-40363-3::foo:head [call lock.lock in=60b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) 2024-01-21T03:22:00.116 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:22:00.124+0000 12a57640 -1 osd.0 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:22:00.746 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:00.755+0000 12a57640 -1 osd.2 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4401.0:4 13.1b 13:dd8ca106:test-rados-api-smithi007-40363-3::foo:head [call lock.lock in=60b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) 2024-01-21T03:22:01.256 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: Running main() from gmock_main.cc 2024-01-21T03:22:01.256 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-01-21T03:22:01.256 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [----------] Global test environment set-up. 2024-01-21T03:22:01.256 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-01-21T03:22:01.256 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-01-21T03:22:01.256 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolList (7801 ms) 2024-01-21T03:22:01.256 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-01-21T03:22:01.256 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (4589 ms) 2024-01-21T03:22:01.256 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-01-21T03:22:01.256 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (4061 ms) 2024-01-21T03:22:01.256 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-01-21T03:22:01.256 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (5868 ms) 2024-01-21T03:22:01.256 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3805 ms) 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (11188 ms) 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6575 ms) 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5981 ms) 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (10862 ms) 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [----------] 9 tests from LibRadosPools (60731 ms total) 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [----------] Global test environment tear-down 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (60731 ms total) 2024-01-21T03:22:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ PASSED ] 9 tests. 2024-01-21T03:22:01.286 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:22:01.295+0000 12a57640 -1 osd.0 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:22:02.108 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:02.033+0000 12a57640 -1 osd.2 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4401.0:4 13.1b 13:dd8ca106:test-rados-api-smithi007-40363-3::foo:head [call lock.lock in=60b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) 2024-01-21T03:22:02.672 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:22:02.652+0000 12a57640 -1 osd.0 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:22:02.934 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:22:02.910+0000 12a57640 -1 osd.6 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4798.0:2 57.15 57:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-01-21T03:22:03.234 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:03.155+0000 12a57640 -1 osd.2 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4401.0:4 13.1b 13:dd8ca106:test-rados-api-smithi007-40363-3::foo:head [call lock.lock in=60b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) 2024-01-21T03:22:03.342 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:22:03.343+0000 12a57640 -1 osd.1 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:3 7.0 7.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) 2024-01-21T03:22:03.913 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:22:03.863+0000 12a57640 -1 osd.0 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:22:04.152 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:22:04.113+0000 12a57640 -1 osd.6 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4798.0:2 57.15 57:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-01-21T03:22:04.191 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:22:04.045+0000 12a57640 -1 osd.3 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4428.0:1 39.0s0 39.db79b668 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T03:22:04.245 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:18.575 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:04.349 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:18.678 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:04.367 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:18.697 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:04.401 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:18.676 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:04.409 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:18.685 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:04.413 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:18.742 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:04.603 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:18.933 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:04.611 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:18.886 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:04.876 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:22:04.740+0000 12a57640 -1 osd.1 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:3 7.0 7.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) 2024-01-21T03:22:04.926 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:19.256 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:04.954 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:22:04.876+0000 12a57640 -1 osd.0 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:22:05.122 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:22:05.109+0000 12a57640 -1 osd.3 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4428.0:1 39.0s0 39:166d9edb:test-rados-api-smithi007-40720-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T03:22:05.672 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:05.677+0000 12a57640 -1 osd.2 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:05.914 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:22:05.892+0000 12a57640 -1 osd.0 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:22:06.137 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:22:06.139+0000 12a57640 -1 osd.3 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4428.0:1 39.0s0 39:166d9edb:test-rados-api-smithi007-40720-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T03:22:06.880 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:22:06.891+0000 12a57640 -1 osd.0 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:22:06.982 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:06.985+0000 12a57640 -1 osd.2 71 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:07.163 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:22:07.164+0000 12a57640 -1 osd.3 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4428.0:1 39.0s0 39:166d9edb:test-rados-api-smithi007-40720-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T03:22:08.113 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:22:08.101+0000 12a57640 -1 osd.0 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:22:08.537 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:22:08.522+0000 12a57640 -1 osd.3 72 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4428.0:1 39.0s0 39:166d9edb:test-rados-api-smithi007-40720-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T03:22:08.603 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:08.551+0000 12a57640 -1 osd.2 72 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:09.269 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:22:09.232+0000 12a57640 -1 osd.0 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4364.0:3 8.9 8:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) 2024-01-21T03:22:09.801 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:22:09.681+0000 12a57640 -1 osd.3 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:3 60.15 60.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e48) 2024-01-21T03:22:10.940 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:22:10.888+0000 12a57640 -1 osd.3 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:3 60.15 60.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e48) 2024-01-21T03:22:18.043 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:22:17.966+0000 12a57640 -1 osd.3 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4837.0:3 65.14 65.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e50) 2024-01-21T03:22:19.019 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:22:19.025+0000 12a57640 -1 osd.3 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4837.0:3 65.14 65.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e50) 2024-01-21T03:22:24.428 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:38.758 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:24.446 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:38.776 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:24.478 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:38.808 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:24.489 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:38.819 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:24.504 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:38.779 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:24.505 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:38.780 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:24.597 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:38.927 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:24.626 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:38.901 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:24.629 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:38.959 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:26.547 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-01-21T03:22:26.547 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-01-21T03:22:26.547 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-01-21T03:22:26.547 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-01-21T03:22:26.547 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (85829 ms) 2024-01-21T03:22:26.547 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (85829 ms total) 2024-01-21T03:22:26.547 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: 2024-01-21T03:22:26.547 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-01-21T03:22:26.547 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (85829 ms total) 2024-01-21T03:22:26.547 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-01-21T03:22:31.835 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:31.755+0000 12a57640 -1 osd.2 95 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:33.421 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:33.361+0000 12a57640 -1 osd.2 95 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:34.704 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:34.615+0000 12a57640 -1 osd.2 96 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:36.124 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:35.987+0000 12a57640 -1 osd.2 99 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:37.140 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:37.057+0000 12a57640 -1 osd.2 99 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:38.105 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:38.053+0000 12a57640 -1 osd.2 100 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:39.069 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:39.076+0000 12a57640 -1 osd.2 101 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:40.045 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:40.054+0000 12a57640 -1 osd.2 101 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:41.123 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:41.105+0000 12a57640 -1 osd.2 103 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:42.192 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:42.164+0000 12a57640 -1 osd.2 103 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:43.311 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:43.139+0000 12a57640 -1 osd.2 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:44.454 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:44.420+0000 12a57640 -1 osd.2 106 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:46.214 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:46.184+0000 12a57640 -1 osd.2 108 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:47.433 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:47.296+0000 12a57640 -1 osd.2 108 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:48.303 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:48.300+0000 12a57640 -1 osd.2 108 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:49.791 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:49.722+0000 12a57640 -1 osd.2 110 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:49.917 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:04.247 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:49.933 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:04.263 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:49.981 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:04:04.256 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:49.990 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:04.320 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:49.998 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:04:04.272 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:50.006 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:04.336 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:50.183 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:04.513 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:50.207 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:04:04.482 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:50.256 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:04.586 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:22:50.819 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:50.815+0000 12a57640 -1 osd.2 111 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:52.184 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:52.169+0000 12a57640 -1 osd.2 112 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:53.308 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:53.313+0000 12a57640 -1 osd.2 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:54.492 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:54.432+0000 12a57640 -1 osd.2 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:55.928 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:55.690+0000 12a57640 -1 osd.2 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:56.925 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:56.810+0000 12a57640 -1 osd.2 116 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:22:58.372 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:22:58.220+0000 12a57640 -1 osd.2 116 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:23:02.412 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:23:02.423+0000 12a57640 -1 osd.2 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:23:04.455 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:23:04.465+0000 12a57640 -1 osd.2 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:23:06.072 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:23:05.906+0000 12a57640 -1 osd.2 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:23:06.943 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:23:06.953+0000 12a57640 -1 osd.2 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:23:08.939 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:23:08.944+0000 12a57640 -1 osd.2 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:23:09.407 INFO:tasks.ceph.mon.c.smithi007.stderr:2024-01-21T03:23:09.359+0000 ff82640 -1 mon.c@2(peon).paxos(paxos updating c 1..416) lease_expire from mon.0 v2:172.21.15.7:3300/0 is 1.407435298s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2024-01-21T03:23:09.945 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:23:09.949+0000 12a57640 -1 osd.2 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:23:10.945 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:23:10.954+0000 12a57640 -1 osd.2 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:23:11.930 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:23:11.934+0000 12a57640 -1 osd.2 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:23:13.300 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:23:13.059+0000 12a57640 -1 osd.2 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:23:15.747 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:23:15.436+0000 12a57640 -1 osd.2 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:23:16.738 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:23:16.576+0000 12a57640 -1 osd.2 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:23:17.867 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-01-21T03:23:17.867 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-01-21T03:23:17.867 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-01-21T03:23:17.867 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-01-21T03:23:17.867 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-01-21T03:23:17.867 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (594 ms) 2024-01-21T03:23:17.867 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-01-21T03:23:17.867 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (1114 ms) 2024-01-21T03:23:17.867 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-01-21T03:23:17.867 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (135506 ms) 2024-01-21T03:23:17.867 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (137214 ms total) 2024-01-21T03:23:17.867 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: 2024-01-21T03:23:17.868 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-01-21T03:23:17.868 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (137216 ms total) 2024-01-21T03:23:17.868 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-01-21T03:23:18.039 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:23:18.050+0000 12a57640 -1 osd.2 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:2 59.1e 59:7a7d4169:::src_object.1:head [writefull 0~4096 in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41) 2024-01-21T03:23:18.225 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-01-21T03:23:18.225 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-01-21T03:23:18.225 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-01-21T03:23:18.225 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-01-21T03:23:18.225 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-01-21T03:23:18.225 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-01-21T03:23:18.225 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-01-21T03:23:18.225 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (39326 ms) 2024-01-21T03:23:18.225 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-01-21T03:23:18.225 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (25224 ms) 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (4701 ms) 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (45650 ms) 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (1205 ms) 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (1530 ms) 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (76 ms) 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (110 ms) 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (3762 ms) 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (270 ms) 2024-01-21T03:23:18.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (104 ms) 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (198 ms) 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (536 ms) 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (462 ms) 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (423 ms) 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (155 ms) 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (123732 ms total) 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (137500 ms total) 2024-01-21T03:23:18.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-01-21T03:23:22.838 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: Running main() from gmock_main.cc 2024-01-21T03:23:22.838 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-01-21T03:23:22.838 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [----------] Global test environment set-up. 2024-01-21T03:23:22.838 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-01-21T03:23:22.838 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-01-21T03:23:22.838 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1532 ms) 2024-01-21T03:23:22.838 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-01-21T03:23:22.838 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (218 ms) 2024-01-21T03:23:22.838 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-01-21T03:23:22.838 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (136166 ms) 2024-01-21T03:23:22.838 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-01-21T03:23:22.838 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:02.424796+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'RemoteReadssmithi007-41049-2' : 1 ]) 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:04.467149+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'RemoteReadssmithi007-41049-2' : 1 ]) 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:05.907342+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'RemoteReadssmithi007-41049-2' : 1 ]) 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:06.294669+0000 mon.b [INF] from='client.? 172.21.15.7:0/232797457' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi007-40360-6", "tierpool": "test-rados-api-smithi007-40360-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:06.954972+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'RemoteReadssmithi007-41049-2' : 1 ]) 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:08.945814+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'RemoteReadssmithi007-41049-2' : 1 ]) 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:09.950532+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'RemoteReadssmithi007-41049-2' : 1 ]) 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:10.955508+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'RemoteReadssmithi007-41049-2' : 1 ]) 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:11.473268+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi007-40360-6", "tierpool": "test-rados-api-smithi007-40360-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:11.858793+0000 mon.a [INF] from='client.? 172.21.15.7:0/1884460504' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi007-40720-6", "pg_num": 4}]': finished 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:11.935525+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'RemoteReadssmithi007-41049-2' : 1 ]) 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:13.490385+0000 mon.a [INF] from='client.? 172.21.15.7:0/1884460504' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi007-40720-4", "tierpool": "test-rados-api-smithi007-40720-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:14.670954+0000 mon.a [WRN] Health check update: 1 slow ops, oldest one blocked for 80 sec, daemons [osd.1,osd.2] have slow ops. (SLOW_OPS) 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:13.061106+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'RemoteReadssmithi007-41049-2' : 1 ]) 2024-01-21T03:23:22.839 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:15.437824+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'RemoteReadssmithi007-41049-2' : 1 ]) 2024-01-21T03:23:27.027 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:16.294831+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi007-40360-6", "tierpool": "test-rados-api-smithi007-40360-7", "force_nonempty": "--force-nonempty" }]': finished 2024-01-21T03:23:27.027 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:17.134478+0000 mon.a [INF] from='client.? 172.21.15.7:0/2135838685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi007-40276-10"}]: dispatch 2024-01-21T03:23:27.027 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:17.350694+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on back (longest 27365.529ms) (OSD_SLOW_PING_TIME_BACK) 2024-01-21T03:23:27.027 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:17.360734+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on front (longest 21523.044ms) (OSD_SLOW_PING_TIME_FRONT) 2024-01-21T03:23:27.027 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:17.363411+0000 mon.a [WRN] Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-21T03:23:27.027 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:17.882741+0000 mon.a [INF] from='client.? 172.21.15.7:0/2135838685' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi007-40276-10"}]: dispatch 2024-01-21T03:23:27.027 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:16.577701+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'RemoteReadssmithi007-41049-2' : 1 ]) 2024-01-21T03:23:27.028 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:18.052938+0000 osd.2 [WRN] 1 slow requests (by type [ 'commit sent; apply or cleanup' : 1 ] most affected pool [ 'RemoteReadssmithi007-41049-2' : 1 ]) 2024-01-21T03:23:27.028 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:18.262165+0000 mon.b [INF] from='client.? 172.21.15.7:0/232797457' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi007-40360-6", "overlaypool": "test-rados-api-smithi007-40360-7"}]: dispatch 2024-01-21T03:23:27.028 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:18.801597+0000 mon.b [INF] from='client.? 172.21.15.7:0/1375297149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi007-40858-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-01-21T03:23:27.028 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:19.688794+0000 mon.a [WRN] Health check update: 1 slow ops, oldest one blocked for 90 sec, osd.2 has slow ops (SLOW_OPS) 2024-01-21T03:23:27.028 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:20.071415+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi007-40360-6", "overlaypool": "test-rados-api-smithi007-40360-7"}]: dispatch 2024-01-21T03:23:27.028 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:20.169727+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi007-40858-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-01-21T03:23:27.028 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:20.313494+0000 mon.a [INF] from='client.? 172.21.15.7:0/1884460504' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi007-40720-4", "tierpool": "test-rados-api-smithi007-40720-6", "force_nonempty": "--force-nonempty" }]': finished 2024-01-21T03:23:27.028 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:20.636448+0000 mon.c [INF] from='client.? 172.21.15.7:0/309886218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi007-40209-16"}]: dispatch 2024-01-21T03:23:27.028 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:20.953447+0000 mon.c [INF] from='client.? 172.21.15.7:0/4235730906' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi007-40238-22"}]: dispatch 2024-01-21T03:23:27.028 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:21.317712+0000 client.admin [INF] onexx 2024-01-21T03:23:27.028 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:21.393748+0000 mon.a [INF] from='client.? 172.21.15.7:0/2748104344' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-01-21T03:23:32.551 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:21.613606+0000 mon.a [INF] from='client.? 172.21.15.7:0/2135838685' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi007-40276-10" cmd: Running main() from gmock_main.cc 2024-01-21T03:23:32.551 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-01-21T03:23:32.551 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: [----------] Global test environment set-up. 2024-01-21T03:23:32.551 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-01-21T03:23:32.551 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-01-21T03:23:32.551 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (5791 ms) 2024-01-21T03:23:32.551 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-01-21T03:23:32.551 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (29752 ms) 2024-01-21T03:23:32.552 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-01-21T03:23:32.552 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (116188 ms) 2024-01-21T03:23:32.552 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: [----------] 3 tests from NeoRadosCmd (151731 ms total) 2024-01-21T03:23:32.552 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: 2024-01-21T03:23:32.552 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: [----------] Global test environment tear-down 2024-01-21T03:23:32.552 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: [==========] 3 tests from 1 test suite ran. (151731 ms total) 2024-01-21T03:23:32.552 INFO:tasks.workunit.client.0.smithi007.stdout: cmd: [ PASSED ] 3 tests. 2024-01-21T03:23:33.506 INFO:tasks.workunit.client.0.smithi007.stdout: cls: Running main() from gmock_main.cc 2024-01-21T03:23:33.507 INFO:tasks.workunit.client.0.smithi007.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-01-21T03:23:33.507 INFO:tasks.workunit.client.0.smithi007.stdout: cls: [----------] Global test environment set-up. 2024-01-21T03:23:33.507 INFO:tasks.workunit.client.0.smithi007.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-01-21T03:23:33.507 INFO:tasks.workunit.client.0.smithi007.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-01-21T03:23:33.507 INFO:tasks.workunit.client.0.smithi007.stdout: cls: [ OK ] NeoRadosCls.DNE (20818 ms) 2024-01-21T03:23:33.507 INFO:tasks.workunit.client.0.smithi007.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-01-21T03:23:33.507 INFO:tasks.workunit.client.0.smithi007.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (131833 ms) 2024-01-21T03:23:33.507 INFO:tasks.workunit.client.0.smithi007.stdout: cls: [----------] 2 tests from NeoRadosCls (152651 ms total) 2024-01-21T03:23:33.507 INFO:tasks.workunit.client.0.smithi007.stdout: cls: 2024-01-21T03:23:33.507 INFO:tasks.workunit.client.0.smithi007.stdout: cls: [----------] Global test environment tear-down 2024-01-21T03:23:33.507 INFO:tasks.workunit.client.0.smithi007.stdout: cls: [==========] 2 tests from 1 test suite ran. (152651 ms total) 2024-01-21T03:23:33.507 INFO:tasks.workunit.client.0.smithi007.stdout: cls: [ PASSED ] 2 tests. 2024-01-21T03:23:35.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-01-21T03:23:35.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] Global test environment set-up. 2024-01-21T03:23:35.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] 7 tests from LibRadosList 2024-01-21T03:23:35.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-01-21T03:23:35.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjects (120942 ms) 2024-01-21T03:23:35.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-01-21T03:23:35.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (1066 ms) 2024-01-21T03:23:35.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-01-21T03:23:35.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo1,foo2,foo3 2024-01-21T03:23:35.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo1 2024-01-21T03:23:35.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo2 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo3 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo1,foo4,foo5 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo4 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo5 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo1 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo6,foo7 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo7 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo6 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo4 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo5 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns2:foo7 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns2:foo6 2024-01-21T03:23:35.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo1 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo1 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo2 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo3 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (2387 ms) 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 1 0 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 10 0 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 13 0 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 7 0 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 14 0 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 0 0 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 15 0 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 11 0 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 5 0 2024-01-21T03:23:35.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 8 0 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 6 0 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 3 0 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 4 0 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 12 0 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 9 0 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 2 0 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (16649 ms) 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: x cursor=MIN 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=1 cursor=7:02547ec2:::1:head 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=10 cursor=7:52ea6a34:::10:head 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=13 cursor=7:566253c9:::13:head 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=7 cursor=7:5c6b0b28:::7:head 2024-01-21T03:23:35.462 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=14 cursor=7:62a1935d:::14:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=0 cursor=7:6cac518f:::0:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=15 cursor=7:863748b0:::15:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=11 cursor=7:89d3ae78:::11:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=5 cursor=7:b29083e3:::5:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=8 cursor=7:bd63b0f1:::8:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=6 cursor=7:c4fdafeb:::6:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=3 cursor=7:cfc208b3:::3:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=4 cursor=7:d83876eb:::4:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=12 cursor=7:de5d7c5f:::12:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=9 cursor=7:e960b815:::9:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=2 cursor=7:f905c69b:::2:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: FIRST> seek to MIN oid=1 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=1 cursor=7:02547ec2:::1:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:02547ec2:::1:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:02547ec2:::1:head -> 1 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=10 cursor=7:52ea6a34:::10:head 2024-01-21T03:23:35.463 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:52ea6a34:::10:head 2024-01-21T03:23:35.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2024-01-21T03:23:35.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=13 cursor=7:566253c9:::13:head 2024-01-21T03:23:35.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:566253c9:::13:head 2024-01-21T03:23:35.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:566253c9:::13:head -> 13 2024-01-21T03:23:35.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=7 cursor=7:5c6b0b28:::7:head 2024-01-21T03:23:35.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2024-01-21T03:23:35.464 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2024-01-21T03:23:36.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=14 cursor=7:62a1935d:::1, "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-01-21T03:23:36.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:21.857405+0000 mon.a [INF] from='client.? 172.21.15.7:0/1884460504' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi007-40720-4", "overlaypool": "test-rados-api-smithi007-40720-6"}]: dispatch 2024-01-21T03:23:36.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:22.365121+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi007-40209-16"}]: dispatch 2024-01-21T03:23:36.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:22.370990+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi007-40238-22"}]: dispatch 2024-01-21T03:23:36.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:21.536494+0000 mon.b [INF] from='client.? 172.21.15.7:0/3093647558' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi007-40363-10"}]: dispatch 2024-01-21T03:23:36.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:23.861048+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi007-40363-10"}]: dispatch 2024-01-21T03:23:36.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:24.125430+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi007-40360-6", "overlaypool": "test-rados-api-smithi007-40360-7"}]': finished 2024-01-21T03:23:36.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:24.160307+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi007-40858-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-01-21T03:23:36.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:24.911378+0000 mon.a [WRN] Health check update: 1 slow ops, oldest one blocked for 95 sec, osd.2 has slow ops (SLOW_OPS) 2024-01-21T03:23:36.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:25.180325+0000 mon.a [WRN] Health check update: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2024-01-21T03:23:36.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:25.190575+0000 mon.a [WRN] Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-21T03:23:36.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:26.171400+0000 mon.c [INF] from='client.? 172.21.15.7:0/4235730906' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi007-40238-22"}]: dispatch 2024-01-21T03:23:36.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:26.178249+0000 mon.c [INF] from='client.? 172.21.15.7:0/309886218' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi007-40209-16"}]: dispatch 2024-01-21T03:23:36.699 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:26.651636+0000 mon.a [INF] from='client.? 172.21.15.7:0/2748104344' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-01-21T03:23:36.699 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:27.359694+0000 client.admin [INF] twoxx 2024-01-21T03:23:36.699 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:27.376418+0000 mon.a [INF] from='client.? 172.21.15.7:0/2748104344' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-01-21T03:23:36.699 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:27.526453+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi007-40238-22"}]: dispatch 2024-01-21T03:23:36.699 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:27.541358+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi007-40209-16"}]: dispatch 2024-01-21T03:23:36.699 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:25.904689+0000 mon.b [INF] from='client.? 172.21.15.7:0/3093647558' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi007-40363-10"}]: dispatch 2024-01-21T03:23:37.915 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:26.012737+0000 mon.b [INF] from='client.? 172.21.15.7:0/2758660628' entity='client.admin' cmd=[{"prefix": "osd era4:head 2024-01-21T03:23:37.915 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:62a1935d:::14:head 2024-01-21T03:23:37.915 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:62a1935d:::14:head -> 14 2024-01-21T03:23:37.915 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=0 cursor=7:6cac518f:::0:head 2024-01-21T03:23:37.915 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:6cac518f:::0:head 2024-01-21T03:23:37.915 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:6cac518f:::0:head -> 0 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=15 cursor=7:863748b0:::15:head 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:863748b0:::15:head 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:863748b0:::15:head -> 15 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=11 cursor=7:89d3ae78:::11:head 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:89d3ae78:::11:head 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=5 cursor=7:b29083e3:::5:head 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:b29083e3:::5:head 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:b29083e3:::5:head -> 5 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=8 cursor=7:bd63b0f1:::8:head 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=6 cursor=7:c4fdafeb:::6:head 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=3 cursor=7:cfc208b3:::3:head 2024-01-21T03:23:37.916 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:cfc208b3:::3:head 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=4 cursor=7:d83876eb:::4:head 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:d83876eb:::4:head 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:d83876eb:::4:head -> 4 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=12 cursor=7:de5d7c5f:::12:head 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=9 cursor=7:e960b815:::9:head 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:e960b815:::9:head 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:e960b815:::9:head -> 9 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=2 cursor=7:f905c69b:::2:head 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:f905c69b:::2:head 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:f905c69b:::2:head -> 2 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:566253c9:::13:head 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:566253c9:::13:head expected=7:566253c9:::13:head 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:566253c9:::13:head -> 13 2024-01-21T03:23:37.917 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=13 expected=13 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:52ea6a34:::10:head 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:52ea6a34:::10:head expected=7:52ea6a34:::10:head 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=10 expected=10 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:bd63b0f1:::8:head expected=7:bd63b0f1:::8:head 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=8 expected=8 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:b29083e3:::5:head 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:b29083e3:::5:head expected=7:b29083e3:::5:head 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:b29083e3:::5:head -> 5 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=5 expected=5 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:89d3ae78:::11:head 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:89d3ae78:::11:head expected=7:89d3ae78:::11:head 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=11 expected=11 2024-01-21T03:23:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:863748b0:::15:head 2024-01-21T03:23:37.919 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:863748b0:::15:head expected=7:863748b0:::15:head 2024-01-21T03:23:37.919 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:863748b0:::15:head -> 15 2024-01-21T03:23:37.919 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=15 expected=15 2024-01-21T03:23:37.919 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2024-01-21T03:23:37.919 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:5c6b0b28:::7:head expected=7:5c6b0b28:::7:head 2024-01-21T03:23:37.919 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2024-01-21T03:23:37.919 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=7 expected=7 2024-01-21T03:23:37.919 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:d83876eb:::4:head 2024-01-21T03:23:37.919 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:d83876eb:::4:head expected=7:d83876eb:::4:head 2024-01-21T03:23:37.919 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:d83876eb:::4:head -> 4 2024-01-21T03:23:37.919 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=4 expected=4 2024-01-21T03:23:37.919 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:cfc208b3:::3:head 2024-01-21T03:23:40.570 INFO:tasks.workunit.client.0.smithi007.stdout: sure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi007-40543-7"}]: dispatch 2024-01-21T03:23:40.570 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:27.984395+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi007-40363-10"}]: dispatch 2024-01-21T03:23:40.570 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:28.002968+0000 mon.a [INF] from='client.? 172.21.15.7:0/2135838685' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi007-40276-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-01-21T03:23:40.570 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:28.006006+0000 mon.a [INF] from='client.? 172.21.15.7:0/1884460504' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi007-40720-4", "overlaypool": "test-rados-api-smithi007-40720-6"}]': finished 2024-01-21T03:23:40.570 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:28.539925+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi007-40543-7"}]: dispatch 2024-01-21T03:23:40.570 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:28.548138+0000 mon.a [INF] from='client.? 172.21.15.7:0/2135838685' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi007-40276-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi007-40276-10"}]: dispatch 2024-01-21T03:23:40.570 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:29.005398+0000 mon.a [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive) 2024-01-21T03:23:40.570 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:29.006248+0000 mon.a [INF] Health check cleared: SLOW_OPS (was: 1 slow ops, oldest one blocked for 101 sec, osd.2 has slow ops) 2024-01-21T03:23:40.570 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:29.397488+0000 mon.a [INF] from='client.? 172.21.15.7:0/1884460504' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi007-40720-6", "mode": "writeback"}]: dispatch 2024-01-21T03:23:40.571 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:29.596746+0000 mon.a [INF] from='client.? 172.21.15.7:0/2748104344' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-01-21T03:23:40.571 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:28.999753+0000 mon.b [INF] from='client.? 172.21.15.7:0/232797457' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi007-40360-6"}]: dispatch 2024-01-21T03:23:40.571 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:29.399860+0000 mon.b [INF] from='client.? 172.21.15.7:0/3093647558' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi007-40363-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-01-21T03:23:40.571 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:29.491410+0000 mon.b [INF] from='client.? 172.21.15.7:0/2758660628' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi007-40543-7"}]: dispatch 2024-01-21T03:23:40.571 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:29.564793+0000 mon.c [INF] from='client.? 172.21.15.7:0/4235730906' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi007-40238-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-01-21T03:23:40.571 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:29.576853+0000 mon.c [INF] from='client.? 172.21.15.7:0/309886218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi007-40209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-01-21T03:23:40.571 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:30.289654+0000 mon.c [INF] from='client.? 172.21.15.7:0/375709152' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi007-40644-7"}]: dispatch 2024-01-21T03:23:50.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:30.826284+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi007-40360-6"}]: dispatch 2024-01-21T03:23:50.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:30.888145+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi007-40363-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-01-21T03:23:50.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:30.899263+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi007-40543-7"}]: dispatch 2024-01-21T03:23:50.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:30.928603+0000 mon.c [INF] from='client.? 172.21.15.7:0/241012637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi007-40724-12"}]: dispatch 2024-01-21T03:23:50.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:31.238922+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi007-40238-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-01-21T03:23:50.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:31.828675+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi007-40209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-01-21T03:23:50.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:31.853604+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi007-40644-7"}]: dispatch 2024-01-21T03:23:50.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:32.147038+0000 client.admin [INF] threexx 2024-01-21T03:23:50.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:32.159179+0000 mon.a [INF] from='client.? 172.21.15.7:0/2748104344' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-01-21T03:23:50.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:32.289634+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-01-21T03:23:50.169 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:33.536134+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi007-40724-12"}]: dispatch 2024-01-21T03:23:50.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-01-21T03:23:34.982227+0000 mon.a [WRN] Health check update: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-21T03:23:50.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (31647 ms) 2024-01-21T03:23:50.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (169563 ms total) 2024-01-21T03:23:50.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: 2024-01-21T03:23:50.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [----------] Global test environment tear-down 2024-01-21T03:23:50.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (169566 ms total) 2024-01-21T03:23:50.170 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ PASSED ] 4 tests. 2024-01-21T03:23:51.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-01-21T03:23:51.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-01-21T03:23:51.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-01-21T03:23:51.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-01-21T03:23:51.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-01-21T03:23:51.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify 2024-01-21T03:23:51.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (79753 ms) 2024-01-21T03:23:51.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-01-21T03:23:51.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (13259 ms) 2024-01-21T03:23:51.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (93013 ms total) 2024-01-21T03:23:51.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: 2024-01-21T03:23:51.055 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (13753 ms) 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (43484 ms) 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (366 ms) 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (167 ms) 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94750307769792 notify_id 536870912001 notifier_gid 4620 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (296 ms) 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94750307769792 notify_id 541165879298 notifier_gid 4620 2024-01-21T03:23:51.056 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (3959 ms) 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94750307769792 notify_id 541165879297 notifier_gid 4620 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (1300 ms) 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94750307769792 notify_id 541165879297 notifier_gid 4620 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (329 ms) 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94750307769792 notify_id 541165879298 notifier_gid 4620 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (645 ms) 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94750307769792 notify_id 541165879299 notifier_gid 4620 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (421 ms) 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: trying... 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94750307769792 notify_id 541165879298 notifier_gid 4620 2024-01-21T03:23:51.057 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: timed out 2024-01-21T03:23:56.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flu api_stat_pp: Running main() from gmock_main.cc 2024-01-21T03:23:56.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-01-21T03:23:56.975 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: seed 40644 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (29139 ms) 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (19697 ms) 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (11 ms) 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (1791 ms) 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (46017 ms) 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (96655 ms total) 2024-01-21T03:23:56.976 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (15017 ms) 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (7365 ms) 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (2787 ms) 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (7893 ms) 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (33062 ms total) 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (176369 ms total) 2024-01-21T03:23:56.977 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-01-21T03:23:57.885 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[41022]: starting. 2024-01-21T03:23:57.885 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[41022]: creating pool ceph_test_rados_list_parallel.smithi007-40964 2024-01-21T03:23:57.885 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[41022]: created object 0... 2024-01-21T03:23:57.885 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[41022]: created object 25... 2024-01-21T03:23:57.885 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[41022]: created object 49... 2024-01-21T03:23:57.885 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[41022]: finishing. 2024-01-21T03:23:57.885 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[41022]: shutting down. 2024-01-21T03:23:57.885 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[41023]: starting. 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[41023]: listing objects. 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[41023]: listed object 0... 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[41023]: listed object 25... 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[41023]: saw 50 objects 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[41023]: shutting down. 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[69672]: starting. 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[69672]: creating pool ceph_test_rados_list_parallel.smithi007-40964 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[69672]: created object 0... 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[69672]: created object 25... 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[69672]: created object 49... 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[69672]: finishing. 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[69672]: shutting down. 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:23:57.886 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[69673]: starting. 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[69673]: listing objects. 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[69673]: listed object 0... 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[69673]: listed object 25... 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[69673]: saw 47 objects 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[69673]: shutting down. 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[69674]: starting. 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[69674]: removed 25 objects... 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[69674]: removed half of the objects 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[69674]: removed 50 objects... 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[69674]: removed 50 objects 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[69674]: shutting down. 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[81207]: starting. 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[81207]: creating pool ceph_test_rados_list_parallel.smithi007-40964 2024-01-21T03:23:57.887 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[81207]: created object 0... 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[81207]: created object 25... 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[81207]: created object 49... 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[81207]: finishing. 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[81207]: shutting down. 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[81208]: starting. 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[81208]: listing objects. 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[81208]: listed object 0... 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[81208]: listed object 25... 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[81208]: listed object 50... 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[81208]: saw 54 objects 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[81208]: shutting down. 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:23:57.888 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:23:57.889 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[81209]: starting. 2024-01-21T03:23:57.889 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[81209]: added 25 objects... 2024-01-21T03:23:57.889 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[81209]: added half of the objects 2024-01-21T03:23:57.889 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[81209]: added 50 objects... 2024-01-21T03:23:57.889 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[81209]: added 50 objects 2024-01-21T03:23:57.889 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[81209]: shutting down. 2024-01-21T03:23:57.889 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:23:57.889 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:24:06.263 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ********************** open_pools_parallel: process_1_[40979]: starting. 2024-01-21T03:24:06.263 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[40979]: creating pool ceph_test_rados_open_pools_parallel.smithi007-40926 2024-01-21T03:24:06.263 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[40979]: created object 0... 2024-01-21T03:24:06.263 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[40979]: created object 25... 2024-01-21T03:24:06.264 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[40979]: created object 49... 2024-01-21T03:24:06.264 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[40979]: finishing. 2024-01-21T03:24:06.264 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[40979]: shutting down. 2024-01-21T03:24:06.264 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_2_[40980]: starting. 2024-01-21T03:24:06.264 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_2_[40980]: rados_pool_create. 2024-01-21T03:24:06.264 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_2_[40980]: rados_ioctx_create. 2024-01-21T03:24:06.264 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_2_[40980]: shutting down. 2024-01-21T03:24:06.264 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******************************* 2024-01-21T03:24:06.264 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[67763]: starting. 2024-01-21T03:24:06.264 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[67763]: creating pool ceph_test_rados_open_pools_parallel.smithi007-40926 2024-01-21T03:24:06.264 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[67763]: created object 0... 2024-01-21T03:24:06.264 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[67763]: created object 25... 2024-01-21T03:24:06.264 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[67763]: created object 49... 2024-01-21T03:24:06.265 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[67763]: finishing. 2024-01-21T03:24:06.265 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[67763]: shutting down. 2024-01-21T03:24:06.265 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******************************* 2024-01-21T03:24:06.265 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_4_[67764]: starting. 2024-01-21T03:24:06.265 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_4_[67764]: rados_pool_create. 2024-01-21T03:24:06.265 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_4_[67764]: rados_ioctx_create. 2024-01-21T03:24:06.265 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_4_[67764]: shutting down. 2024-01-21T03:24:06.265 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******************************* 2024-01-21T03:24:06.265 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******************************* 2024-01-21T03:24:06.265 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-01-21T03:24:07.219 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[41026]: starting. 2024-01-21T03:24:07.219 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[41026]: creating pool ceph_test_rados_delete_pools_parallel.smithi007-40970 2024-01-21T03:24:07.220 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[41026]: created object 0... 2024-01-21T03:24:07.220 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[41026]: created object 25... 2024-01-21T03:24:07.220 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[41026]: created object 49... 2024-01-21T03:24:07.220 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[41026]: finishing. 2024-01-21T03:24:07.220 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[41026]: shutting down. 2024-01-21T03:24:07.220 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_2_[41028]: starting. 2024-01-21T03:24:07.220 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_2_[41028]: deleting pool ceph_test_rados_delete_pools_parallel.smithi007-40970 2024-01-21T03:24:07.220 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_2_[41028]: shutting down. 2024-01-21T03:24:07.220 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2024-01-21T03:24:07.220 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[67246]: starting. 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[67246]: creating pool ceph_test_rados_delete_pools_parallel.smithi007-40970 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[67246]: created object 0... 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[67246]: created object 25... 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[67246]: created object 49... 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[67246]: finishing. 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[67246]: shutting down. 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[67248]: starting. 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[67248]: listing objects. 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[67248]: listed object 0... 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[67248]: listed object 25... 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[67248]: saw 50 objects 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[67248]: shutting down. 2024-01-21T03:24:07.221 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2024-01-21T03:24:07.222 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_4_[67247]: starting. 2024-01-21T03:24:07.222 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_4_[67247]: deleting pool ceph_test_rados_delete_pools_parallel.smithi007-40970 2024-01-21T03:24:07.222 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_4_[67247]: shutting down. 2024-01-21T03:24:07.222 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2024-01-21T03:24:07.222 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2024-01-21T03:24:07.222 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-01-21T03:24:08.382 INFO:tasks.workunit.client.0.smithi007.stdout:1a042919c0 2024-01-21T03:24:08.382 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a042924b0 2024-01-21T03:24:08.382 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a04292fa0 2024-01-21T03:24:08.382 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a04293a90 2024-01-21T03:24:08.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a04294580 2024-01-21T03:24:08.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a04295070 2024-01-21T03:24:08.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03957900 2024-01-21T03:24:08.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03958300 2024-01-21T03:24:08.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03958d00 2024-01-21T03:24:08.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03952b40 2024-01-21T03:24:08.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03952cb0 2024-01-21T03:24:08.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a042936e0 2024-01-21T03:24:08.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a04293890 2024-01-21T03:24:08.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a042941d0 2024-01-21T03:24:08.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a042943a0 2024-01-21T03:24:08.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03957590 2024-01-21T03:24:08.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03957f50 2024-01-21T03:24:08.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a03958190 2024-01-21T03:24:08.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a04292de0 2024-01-21T03:24:08.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x561a042917c0 2024-01-21T03:24:08.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: done. 2024-01-21T03:24:08.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (47589 ms) 2024-01-21T03:24:08.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (47589 ms total) 2024-01-21T03:24:08.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2024-01-21T03:24:08.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-01-21T03:24:08.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-01-21T03:24:08.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-01-21T03:24:08.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-01-21T03:24:08.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.Exec (20538 ms) 2024-01-21T03:24:08.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (27978 ms) 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (4697 ms) 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.Applications (6380 ms) 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (73725 ms) 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 8 tests from LibRadosMisc (133318 ms total) 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2024-01-21T03:24:08.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] Global test environment tear-down 2024-01-21T03:24:08.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (187860 ms total) 2024-01-21T03:24:08.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ PASSED ] 12 tests. 2024-01-21T03:24:09.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: Running main() from gmock_main.cc 2024-01-21T03:24:09.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-01-21T03:24:09.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] Global test environment set-up. 2024-01-21T03:24:09.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-01-21T03:24:09.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-01-21T03:24:09.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (52438 ms) 2024-01-21T03:24:09.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-01-21T03:24:09.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockShared (13123 ms) 2024-01-21T03:24:09.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-01-21T03:24:09.881 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (7989 ms) 2024-01-21T03:24:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-01-21T03:24:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (4020 ms) 2024-01-21T03:24:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-01-21T03:24:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (8417 ms) 2024-01-21T03:24:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-01-21T03:24:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.Unlock (15719 ms) 2024-01-21T03:24:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-01-21T03:24:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (8508 ms) 2024-01-21T03:24:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-01-21T03:24:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (8226 ms) 2024-01-21T03:24:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] 8 tests from LibRadosLock (118440 ms total) 2024-01-21T03:24:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: 2024-01-21T03:24:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (14536 ms) 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (7568 ms) 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1304 ms) 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1271 ms) 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (218 ms) 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (187 ms) 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-01-21T03:24:09.883 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (214 ms) 2024-01-21T03:24:09.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-01-21T03:24:09.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (218 ms) 2024-01-21T03:24:09.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (25516 ms total) 2024-01-21T03:24:09.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: 2024-01-21T03:24:09.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] Global test environment tear-down 2024-01-21T03:24:09.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (189436 ms total) 2024-01-21T03:24:09.884 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ PASSED ] 16 tests. 2024-01-21T03:24:11.593 INFO:tasks.workunit.client.0.smithi007.stdout:shing 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushed 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3211 ms) 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: trying... 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94750307769792 notify_id 545460846596 notifier_gid 4620 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: timed out 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushing 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushed 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3549 ms) 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: List watches 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify2 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94750307769792 notify_id 545460846594 notifier_gid 4620 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify2 done 2024-01-21T03:24:11.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: watch_check 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: unwatch2 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushing 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: done 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (5280 ms) 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: List watches 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify2 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 94750307769792 notify_id 549755813891 notifier_gid 4620 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify2 done 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: watch_check 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: unwatch2 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushing 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: done 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3230 ms) 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (79991 ms total) 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: 2024-01-21T03:24:11.595 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-01-21T03:24:11.596 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (190960 ms total) 2024-01-21T03:24:11.596 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-01-21T03:24:21.387 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:35.718 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:21.456 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:35.786 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:21.462 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:35.737 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:21.465 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:35.741 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:21.470 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:35.800 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:21.475 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:35.806 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:21.705 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:36.036 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:21.715 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:35.991 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:21.723 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:36.053 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:28.957 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify_test_cb 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (48256 ms) 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94721895980096 err -107 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (34719 ms) 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94721895991504 err -107 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (9916 ms) 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4540 notify_id 493921239040 cookie 94721895993632 2024-01-21T03:24:28.958 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (19063 ms) 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4540 notify_id 498216206336 cookie 94721896010448 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (1665 ms) 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4540 notify_id 502511173633 cookie 94721896010448 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (5372 ms) 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4540 notify_id 506806140929 cookie 94721896010448 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4540 notify_id 506806140929 cookie 94721896012496 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (1266 ms) 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4540 notify_id 506806140930 cookie 94721896010448 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4540 notify_id 506806140931 cookie 94721896010448 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3138 ms) 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-01-21T03:24:28.959 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94721896014816 err -107 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4540 notify_id 511101108225 cookie 94721896014816 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (6941 ms) 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94721896014816 err -107 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (6500 ms) 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (136837 ms total) 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify_test_cb 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (17513 ms) 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (17513 ms total) 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-01-21T03:24:28.960 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (208306 ms total) 2024-01-21T03:24:28.961 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-01-21T03:24:39.109 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: Running main() from gmock_main.cc 2024-01-21T03:24:39.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-01-21T03:24:39.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] Global test environment set-up. 2024-01-21T03:24:39.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-01-21T03:24:39.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: seed 40238 2024-01-21T03:24:39.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-01-21T03:24:39.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-01-21T03:24:39.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-01-21T03:24:39.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (44507 ms) 2024-01-21T03:24:39.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-01-21T03:24:39.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (19144 ms) 2024-01-21T03:24:39.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-01-21T03:24:39.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (49611 ms) 2024-01-21T03:24:39.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-01-21T03:24:39.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (1090 ms) 2024-01-21T03:24:39.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-01-21T03:24:39.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (1184 ms) 2024-01-21T03:24:39.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-01-21T03:24:39.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (701 ms) 2024-01-21T03:24:39.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-01-21T03:24:39.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (108 ms) 2024-01-21T03:24:39.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-01-21T03:24:39.112 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (285 ms) 2024-01-21T03:24:39.112 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-01-21T03:24:39.112 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (118 ms) 2024-01-21T03:24:39.112 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-01-21T03:24:39.112 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (57 ms) 2024-01-21T03:24:39.112 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-01-21T03:24:39.112 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (79 ms) 2024-01-21T03:24:39.112 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-01-21T03:24:39.112 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (66 ms) 2024-01-21T03:24:39.112 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-01-21T03:24:39.112 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (4103 ms) 2024-01-21T03:24:39.112 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-01-21T03:24:39.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (1204 ms) 2024-01-21T03:24:39.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-01-21T03:24:39.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (608 ms) 2024-01-21T03:24:39.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-01-21T03:24:39.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (208 ms) 2024-01-21T03:24:39.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-01-21T03:24:39.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (184 ms) 2024-01-21T03:24:39.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-01-21T03:24:39.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (53 ms) 2024-01-21T03:24:39.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-01-21T03:24:39.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (159 ms) 2024-01-21T03:24:39.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (123469 ms total) 2024-01-21T03:24:39.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: 2024-01-21T03:24:39.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-01-21T03:24:39.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-01-21T03:24:39.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (12596 ms) 2024-01-21T03:24:39.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-01-21T03:24:39.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (16486 ms) 2024-01-21T03:24:39.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-01-21T03:24:39.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (2465 ms) 2024-01-21T03:24:39.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-01-21T03:24:39.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (1577 ms) 2024-01-21T03:24:39.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-01-21T03:24:41.091 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundT api_stat: Running main() from gmock_main.cc 2024-01-21T03:24:41.091 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-01-21T03:24:41.091 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] Global test environment set-up. 2024-01-21T03:24:41.091 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.Stat (40086 ms) 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (33400 ms) 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.StatNS (43577 ms) 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (911 ms) 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (6532 ms) 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] 5 tests from LibRadosStat (124506 ms total) 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-01-21T03:24:41.092 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (24103 ms) 2024-01-21T03:24:41.093 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-01-21T03:24:41.093 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (5990 ms) 2024-01-21T03:24:41.093 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-01-21T03:24:41.093 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (128 ms) 2024-01-21T03:24:41.093 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-01-21T03:24:41.093 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1179 ms) 2024-01-21T03:24:41.093 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (31400 ms total) 2024-01-21T03:24:41.093 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: 2024-01-21T03:24:41.093 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] Global test environment tear-down 2024-01-21T03:24:41.093 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (220519 ms total) 2024-01-21T03:24:41.093 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ PASSED ] 9 tests. 2024-01-21T03:24:41.854 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:56.185 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:41.864 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:56.195 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:41.928 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:56.203 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:41.942 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:56.217 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:41.944 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:56.274 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:41.947 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:56.277 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:42.003 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:56.334 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:42.034 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:56.310 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:42.063 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:56.394 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:24:46.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-01-21T03:24:46.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-01-21T03:24:46.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-01-21T03:24:46.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-01-21T03:24:46.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: seed 40363 2024-01-21T03:24:46.770 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (19545 ms) 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (42575 ms) 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (3575 ms) 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (6279 ms) 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (22321 ms) 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (22020 ms) 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (6020 ms) 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-01-21T03:24:46.771 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (134 ms) 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (122469 ms total) 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (9153 ms) 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (7251 ms) 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (6706 ms) 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (4673 ms) 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (2188 ms) 2024-01-21T03:24:46.772 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-01-21T03:24:46.773 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (3968 ms) 2024-01-21T03:24:46.773 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-01-21T03:24:46.773 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4017 ms) 2024-01-21T03:24:46.773 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-01-21T03:24:46.773 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (2125 ms) 2024-01-21T03:24:46.773 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (40082 ms total) 2024-01-21T03:24:46.773 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: 2024-01-21T03:24:46.773 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-01-21T03:24:46.773 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (226282 ms total) 2024-01-21T03:24:46.773 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-01-21T03:25:01.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: Running main() from gmock_main.cc 2024-01-21T03:25:01.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] Global test environment set-up. 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (65596 ms) 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: no timeout :/ 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: no timeout :/ 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: no timeout :/ 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: no timeout :/ 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: no timeout :/ 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (2329 ms) 2024-01-21T03:25:01.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (31452 ms) 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.Checksum (7264 ms) 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (9698 ms) 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (145 ms) 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (93 ms) 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (54 ms) 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.TruncTest (5623 ms) 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (74 ms) 2024-01-21T03:25:01.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (145 ms) 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.RmXattr (582 ms) 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.XattrIter (169 ms) 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] 14 tests from LibRadosIo (123224 ms total) 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (14906 ms) 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (5945 ms) 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (2244 ms) 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-01-21T03:25:01.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (8432 ms) 2024-01-21T03:25:01.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-01-21T03:25:01.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (11424 ms) 2024-01-21T03:25:01.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-01-21T03:25:01.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (1501 ms) 2024-01-21T03:25:01.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-01-21T03:25:01.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (983 ms) 2024-01-21T03:25:01.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-01-21T03:25:01.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (1540 ms) 2024-01-21T03:25:01.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-01-21T03:25:01.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (7124 ms) 2024-01-21T03:25:01.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-01-21T03:25:01.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (266 ms) 2024-01-21T03:25:01.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] 10 tests from LibRadosIoEC (54365 ms total) 2024-01-21T03:25:01.443 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: 2024-01-21T03:25:01.443 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] Global test environment tear-down 2024-01-21T03:25:01.443 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [==========] 24 tests from 2 test suites ran. (241017 ms total) 2024-01-21T03:25:01.443 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ PASSED ] 24 tests. 2024-01-21T03:25:02.025 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:16.356 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:02.100 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:16.431 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:02.130 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:16.406 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:02.132 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:16.407 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:02.132 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:16.462 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:02.133 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:16.464 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:02.222 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:16.553 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:02.223 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:16.499 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:02.226 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:16.557 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:11.074 INFO:tasks.workunit.client.0.smithi007.stdout:********* 2024-01-21T03:25:11.074 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[94172]: starting. 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[94172]: creating pool ceph_test_rados_list_parallel.smithi007-40964 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[94172]: created object 0... 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[94172]: created object 25... 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[94172]: created object 49... 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[94172]: finishing. 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[94172]: shutting down. 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[94175]: starting. 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[94175]: listing objects. 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[94175]: listed object 0... 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[94175]: listed object 25... 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[94175]: listed object 50... 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[94175]: listed object 75... 2024-01-21T03:25:11.075 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[94175]: saw 99 objects 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[94175]: shutting down. 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[94182]: starting. 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[94182]: removed 25 objects... 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[94182]: removed half of the objects 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[94182]: removed 50 objects... 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[94182]: removed 50 objects 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[94182]: shutting down. 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[94177]: starting. 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[94177]: added 25 objects... 2024-01-21T03:25:11.076 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[94177]: added half of the objects 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[94177]: added 50 objects... 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[94177]: added 50 objects 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[94177]: shutting down. 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[94176]: starting. 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[94176]: added 25 objects... 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[94176]: added half of the objects 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[94176]: added 50 objects... 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[94176]: added 50 objects 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[94176]: shutting down. 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.077 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[103175]: starting. 2024-01-21T03:25:11.078 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[103175]: creating pool ceph_test_rados_list_parallel.smithi007-40964 2024-01-21T03:25:11.078 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[103175]: created object 0... 2024-01-21T03:25:11.078 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[103175]: created object 25... 2024-01-21T03:25:11.078 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[103175]: created object 49... 2024-01-21T03:25:11.078 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[103175]: finishing. 2024-01-21T03:25:11.078 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[103175]: shutting down. 2024-01-21T03:25:11.078 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.078 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.078 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.078 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:11.078 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[103177]: starting. 2024-01-21T03:25:11.078 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[103177]: listing objects. 2024-01-21T03:25:11.078 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[103177]: listed object 0... 2024-01-21T03:25:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[103177]: listripPP2 (1467 ms) 2024-01-21T03:25:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-01-21T03:25:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (1996 ms) 2024-01-21T03:25:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-01-21T03:25:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (3881 ms) 2024-01-21T03:25:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-01-21T03:25:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (1253 ms) 2024-01-21T03:25:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (887 ms) 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (2265 ms) 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (2043 ms) 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (2185 ms) 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (4476 ms) 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (2399 ms) 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (6018 ms) 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (2174 ms) 2024-01-21T03:25:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-01-21T03:25:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (1909 ms) 2024-01-21T03:25:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (66078 ms total) 2024-01-21T03:25:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: 2024-01-21T03:25:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] Global test environment tear-down 2024-01-21T03:25:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (252766 ms total) 2024-01-21T03:25:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-01-21T03:25:22.214 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:36.545 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:22.296 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:36.627 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:22.387 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:36.718 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:22.389 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:36.720 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:22.405 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:36.681 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:22.413 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:36.689 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:22.448 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:36.779 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:22.501 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:36.777 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:22.544 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:36.875 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:26.460 INFO:tasks.workunit.client.0.smithi007.stdout: io: Running main() from gmock_main.cc 2024-01-21T03:25:26.460 INFO:tasks.workunit.client.0.smithi007.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-01-21T03:25:26.460 INFO:tasks.workunit.client.0.smithi007.stdout: io: [----------] Global test environment set-up. 2024-01-21T03:25:26.460 INFO:tasks.workunit.client.0.smithi007.stdout: io: [----------] 14 tests from NeoRadosIo 2024-01-21T03:25:26.460 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-01-21T03:25:26.460 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.Limits (68597 ms) 2024-01-21T03:25:26.460 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-01-21T03:25:26.460 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (35013 ms) 2024-01-21T03:25:26.460 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-01-21T03:25:26.460 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.ReadOp (29888 ms) 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.SparseRead (45873 ms) 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.RoundTrip (18148 ms) 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (7160 ms) 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (4552 ms) 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (19932 ms) 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (9200 ms) 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.Trunc (4971 ms) 2024-01-21T03:25:26.461 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-01-21T03:25:26.462 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.Remove (7532 ms) 2024-01-21T03:25:26.462 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-01-21T03:25:26.462 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (5368 ms) 2024-01-21T03:25:26.462 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-01-21T03:25:26.462 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.RmXattr (6406 ms) 2024-01-21T03:25:26.462 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-01-21T03:25:26.462 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2972 ms) 2024-01-21T03:25:26.462 INFO:tasks.workunit.client.0.smithi007.stdout: io: [----------] 14 tests from NeoRadosIo (265612 ms total) 2024-01-21T03:25:26.462 INFO:tasks.workunit.client.0.smithi007.stdout: io: 2024-01-21T03:25:26.462 INFO:tasks.workunit.client.0.smithi007.stdout: io: [----------] Global test environment tear-down 2024-01-21T03:25:26.462 INFO:tasks.workunit.client.0.smithi007.stdout: io: [==========] 14 tests from 1 test suite ran. (265612 ms total) 2024-01-21T03:25:26.462 INFO:tasks.workunit.client.0.smithi007.stdout: io: [ PASSED ] 14 tests. 2024-01-21T03:25:26.489 INFO:tasks.workunit.client.0.smithi007.stdout:ed object 25... 2024-01-21T03:25:26.489 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[103177]: listed object 50... 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[103177]: listed object 75... 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[103177]: listed object 100... 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[103177]: listed object 125... 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[103177]: saw 150 objects 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[103177]: shutting down. 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[103178]: starting. 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[103178]: added 25 objects... 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[103178]: added half of the objects 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[103178]: added 50 objects... 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[103178]: added 50 objects 2024-01-21T03:25:26.490 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[103178]: shutting down. 2024-01-21T03:25:26.491 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:26.491 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:26.491 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:26.491 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:26.491 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-01-21T03:25:26.491 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******* SUCCESS ********** 2024-01-21T03:25:34.560 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: Running main() from gmock_main.cc 2024-01-21T03:25:34.560 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-01-21T03:25:34.560 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [----------] Global test environment set-up. 2024-01-21T03:25:34.560 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-01-21T03:25:34.560 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-01-21T03:25:34.560 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (55978 ms) 2024-01-21T03:25:34.560 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-01-21T03:25:34.560 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (21905 ms) 2024-01-21T03:25:34.560 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-01-21T03:25:34.560 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (14379 ms) 2024-01-21T03:25:34.560 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (56948 ms) 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (50497 ms) 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (199707 ms total) 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (8267 ms) 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (10091 ms) 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (14097 ms) 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (12125 ms) 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-01-21T03:25:34.561 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (6626 ms) 2024-01-21T03:25:34.562 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-01-21T03:25:34.562 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi007-41379-11. 2024-01-21T03:25:34.562 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: Waiting for snaps to purge. 2024-01-21T03:25:34.562 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (22721 ms) 2024-01-21T03:25:34.562 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (73927 ms total) 2024-01-21T03:25:34.562 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: 2024-01-21T03:25:34.562 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [----------] Global test environment tear-down 2024-01-21T03:25:34.562 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (273634 ms total) 2024-01-21T03:25:34.562 INFO:tasks.workunit.client.0.smithi007.stdout: snapshots: [ PASSED ] 11 tests. 2024-01-21T03:25:35.584 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: Running main() from gmock_main.cc 2024-01-21T03:25:35.584 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-01-21T03:25:35.584 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [----------] Global test environment set-up. 2024-01-21T03:25:35.584 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-01-21T03:25:35.584 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-01-21T03:25:35.584 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (21933 ms) 2024-01-21T03:25:35.584 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-01-21T03:25:35.584 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (66574 ms) 2024-01-21T03:25:35.584 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-01-21T03:25:35.584 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (11346 ms) 2024-01-21T03:25:35.584 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-01-21T03:25:35.584 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (33341 ms) 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (29583 ms) 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (36946 ms) 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (7230 ms) 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (7877 ms) 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (7388 ms) 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (6852 ms) 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (17459 ms) 2024-01-21T03:25:35.585 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-01-21T03:25:35.586 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (12819 ms) 2024-01-21T03:25:35.586 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-01-21T03:25:35.586 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (9264 ms) 2024-01-21T03:25:35.586 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-01-21T03:25:35.586 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (6058 ms) 2024-01-21T03:25:35.586 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (274671 ms total) 2024-01-21T03:25:35.586 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: 2024-01-21T03:25:35.586 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [----------] Global test environment tear-down 2024-01-21T03:25:35.586 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (274671 ms total) 2024-01-21T03:25:35.586 INFO:tasks.workunit.client.0.smithi007.stdout: read_operations: [ PASSED ] 14 tests. 2024-01-21T03:25:40.615 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-01-21T03:25:40.615 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-01-21T03:25:40.615 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-01-21T03:25:40.615 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-01-21T03:25:40.615 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: seed 40502 2024-01-21T03:25:40.615 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (30147 ms) 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (20148 ms) 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (23387 ms) 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (8836 ms) 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (64967 ms) 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (147485 ms total) 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (15505 ms) 2024-01-21T03:25:40.616 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (15263 ms) 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (13972 ms) 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (9505 ms) 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (3479 ms) 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-766-g0edf41a6/rpm/el9/BUILD/ceph-19.0.0-766-g0edf41a6/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi007-40502-7 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: waiting for snaps to purge 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (20873 ms) 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (78597 ms total) 2024-01-21T03:25:40.617 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2024-01-21T03:25:40.618 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-01-21T03:25:40.618 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-01-21T03:25:40.618 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (7 ms) 2024-01-21T03:25:40.618 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-01-21T03:25:40.618 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5225 ms) 2024-01-21T03:25:40.618 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5232 ms total) 2024-01-21T03:25:40.618 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2024-01-21T03:25:40.618 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-01-21T03:25:40.618 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-01-21T03:25:40.618 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3701 ms) 2024-01-21T03:25:40.618 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-01-21T03:25:41.689 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] Lib api_snapshots: Running main() from gmock_main.cc 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] Global test environment set-up. 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (37788 ms) 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (10985 ms) 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (14049 ms) 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (64874 ms) 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (127696 ms total) 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-01-21T03:25:41.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (41493 ms) 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (6777 ms) 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (17446 ms) 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (65717 ms total) 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (5913 ms) 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (4484 ms) 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (6565 ms) 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2970 ms) 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (19932 ms total) 2024-01-21T03:25:41.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: 2024-01-21T03:25:41.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-01-21T03:25:41.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-01-21T03:25:41.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (7877 ms) 2024-01-21T03:25:41.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-01-21T03:25:41.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (5715 ms) 2024-01-21T03:25:41.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (13592 ms total) 2024-01-21T03:25:41.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: 2024-01-21T03:25:41.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] Global test environment tear-down 2024-01-21T03:25:41.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (281118 ms total) 2024-01-21T03:25:41.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-01-21T03:25:42.341 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:56.672 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:42.349 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:56.680 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:42.402 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:56.677 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:42.403 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:56.679 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:42.410 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:56.741 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:42.414 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:56.745 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:42.430 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:56.761 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:42.464 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:56.740 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:25:42.491 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:56.822 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:02.554 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:16.885 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:02.607 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:16.883 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:02.608 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:16.884 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:02.620 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:16.951 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:02.662 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:16.938 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:02.676 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:17.007 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:02.680 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:17.011 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:02.696 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:17.027 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:02.739 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:17.070 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:09.420 INFO:tasks.workunit.client.0.smithi007.stdout:RadosSnapshotsECPP.SnapRemovePP (2363 ms) 2024-01-21T03:26:09.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-01-21T03:26:09.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (3089 ms) 2024-01-21T03:26:09.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-01-21T03:26:09.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2218 ms) 2024-01-21T03:26:09.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (11371 ms total) 2024-01-21T03:26:09.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2024-01-21T03:26:09.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-01-21T03:26:09.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-01-21T03:26:09.420 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (5094 ms) 2024-01-21T03:26:09.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-01-21T03:26:09.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (5195 ms) 2024-01-21T03:26:09.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-01-21T03:26:09.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4274 ms) 2024-01-21T03:26:09.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (14563 ms total) 2024-01-21T03:26:09.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2024-01-21T03:26:09.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-01-21T03:26:09.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (308869 ms total) 2024-01-21T03:26:09.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-01-21T03:26:09.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-01-21T03:26:09.421 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-01-21T03:26:15.755 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:30.086 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:15.760 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:30.091 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:15.794 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:30.125 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:15.819 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:30.095 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:15.821 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:30.097 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:15.843 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:30.174 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:15.868 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:30.144 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:15.895 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:30.226 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:15.917 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:30.248 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:22.606 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:36.936 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:22.614 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:36.945 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:22.645 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:36.976 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:22.670 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:36.946 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:22.673 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:36.949 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:22.675 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:37.005 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:22.678 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:37.009 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:22.695 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:36.971 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:22.703 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:37.034 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:22.821 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:cfc208b3:::3:head expected=7:cfc208b3:::3:head 2024-01-21T03:26:22.821 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2024-01-21T03:26:22.821 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=3 expected=3 2024-01-21T03:26:22.821 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2024-01-21T03:26:22.821 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:c4fdafeb:::6:head expected=7:c4fdafeb:::6:head 2024-01-21T03:26:22.821 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2024-01-21T03:26:22.821 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=6 expected=6 2024-01-21T03:26:22.821 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:02547ec2:::1:head 2024-01-21T03:26:22.821 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:02547ec2:::1:head expected=7:02547ec2:::1:head 2024-01-21T03:26:22.821 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:02547ec2:::1:head -> 1 2024-01-21T03:26:22.821 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=1 expected=1 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:f905c69b:::2:head 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:f905c69b:::2:head expected=7:f905c69b:::2:head 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:f905c69b:::2:head -> 2 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=2 expected=2 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:e960b815:::9:head 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:e960b815:::9:head expected=7:e960b815:::9:head 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:e960b815:::9:head -> 9 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=9 expected=9 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:de5d7c5f:::12:head expected=7:de5d7c5f:::12:head 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=12 expected=12 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:6cac518f:::0:head 2024-01-21T03:26:22.822 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:6cac518f:::0:head expected=7:6cac518f:::0:head 2024-01-21T03:26:22.823 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:6cac518f:::0:head -> 0 2024-01-21T03:26:22.823 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=0 expected=0 2024-01-21T03:26:22.823 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 7:62a1935d:::14:head 2024-01-21T03:26:22.823 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=7:62a1935d:::14:head expected=7:62a1935d:::14:head 2024-01-21T03:26:22.823 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 7:62a1935d:::14:head -> 14 2024-01-21T03:26:22.823 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=14 expected=14 2024-01-21T03:26:22.823 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (13597 ms) 2024-01-21T03:26:22.823 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-01-21T03:26:22.823 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (133106 ms) 2024-01-21T03:26:22.823 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: split 0/5 -> MIN 7:33333333::::head 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: split 1/5 -> 7:33333333::::head 7:66666666::::head 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: split 2/5 -> 7:66666666::::head 7:99999999::::head 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: split 3/5 -> 7:99999999::::head 7:cccccccc::::head 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: split 4/5 -> 7:cccccccc::::head MAX 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (16414 ms) 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] 7 tests from LibRadosList (304164 ms total) 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (2483 ms) 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo1,foo2,foo3 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo1 2024-01-21T03:26:22.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo2 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo3 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo1,foo4,foo5 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo4 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo5 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo1 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo6,foo7 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo7 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo6 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo4 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo5 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns2:foo7 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns2:foo6 2024-01-21T03:26:22.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo1 2024-01-21T03:26:22.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo1 2024-01-21T03:26:22.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo2 2024-01-21T03:26:22.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo3 2024-01-21T03:26:22.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (4093 ms) 2024-01-21T03:26:22.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-01-21T03:26:22.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 1 0 2024-01-21T03:26:22.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 10 0 2024-01-21T03:26:22.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 13 0 2024-01-21T03:26:25.908 INFO:tasks.workunit.client.0.smithi007.stdout: list: Running main() from gmock_main.cc 2024-01-21T03:26:25.909 INFO:tasks.workunit.client.0.smithi007.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-01-21T03:26:25.909 INFO:tasks.workunit.client.0.smithi007.stdout: list: [----------] Global test environment set-up. 2024-01-21T03:26:25.909 INFO:tasks.workunit.client.0.smithi007.stdout: list: [----------] 3 tests from NeoradosList 2024-01-21T03:26:25.909 INFO:tasks.workunit.client.0.smithi007.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-01-21T03:26:25.909 INFO:tasks.workunit.client.0.smithi007.stdout: list: [ OK ] NeoradosList.ListObjects (133575 ms) 2024-01-21T03:26:25.909 INFO:tasks.workunit.client.0.smithi007.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-01-21T03:26:25.909 INFO:tasks.workunit.client.0.smithi007.stdout: list: [ OK ] NeoradosList.ListObjectsNS (51805 ms) 2024-01-21T03:26:25.909 INFO:tasks.workunit.client.0.smithi007.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-01-21T03:26:25.909 INFO:tasks.workunit.client.0.smithi007.stdout: list: [ OK ] NeoradosList.ListObjectsMany (139654 ms) 2024-01-21T03:26:25.909 INFO:tasks.workunit.client.0.smithi007.stdout: list: [----------] 3 tests from NeoradosList (325035 ms total) 2024-01-21T03:26:25.909 INFO:tasks.workunit.client.0.smithi007.stdout: list: 2024-01-21T03:26:25.909 INFO:tasks.workunit.client.0.smithi007.stdout: list: [----------] Global test environment tear-down 2024-01-21T03:26:25.909 INFO:tasks.workunit.client.0.smithi007.stdout: list: [==========] 3 tests from 1 test suite ran. (325035 ms total) 2024-01-21T03:26:25.910 INFO:tasks.workunit.client.0.smithi007.stdout: list: [ PASSED ] 3 tests. 2024-01-21T03:26:30.036 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 7 0 2024-01-21T03:26:30.036 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 14 0 2024-01-21T03:26:30.036 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 0 0 2024-01-21T03:26:30.036 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 15 0 2024-01-21T03:26:30.036 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 11 0 2024-01-21T03:26:30.036 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 5 0 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 8 0 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 6 0 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 3 0 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 4 0 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 12 0 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 9 0 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 2 0 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (3139 ms) 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] 3 tests from LibRadosListEC (9715 ms total) 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2968 ms) 2024-01-21T03:26:30.037 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] 1 test from LibRadosListNP (2968 ms total) 2024-01-21T03:26:30.038 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 2024-01-21T03:26:30.038 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] Global test environment tear-down 2024-01-21T03:26:30.038 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [==========] 11 tests from 3 test suites ran. (329563 ms total) 2024-01-21T03:26:30.038 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ PASSED ] 11 tests. 2024-01-21T03:26:42.907 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:57.238 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:42.911 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:57.242 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:42.973 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:57.304 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:42.979 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:57.255 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:42.980 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:57.311 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:42.984 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:57.315 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:42.988 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:57.264 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:43.042 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:57.373 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:26:43.047 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:57.323 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:02.973 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:17.304 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:02.981 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:17.313 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:03.007 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:17.283 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:03.011 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:17.287 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:03.041 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:17.372 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:03.055 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:17.331 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:03.062 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:17.393 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:03.065 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:17.396 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:03.119 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:17.450 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:10.013 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: Running main() from gmock_main.cc 2024-01-21T03:27:10.013 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-01-21T03:27:10.013 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: [----------] Global test environment set-up. 2024-01-21T03:27:10.013 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-01-21T03:27:10.014 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-01-21T03:27:10.014 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (86596 ms) 2024-01-21T03:27:10.014 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-01-21T03:27:10.014 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (99709 ms) 2024-01-21T03:27:10.014 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-01-21T03:27:10.014 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (182824 ms) 2024-01-21T03:27:10.014 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: [----------] 3 tests from NeoradosECList (369129 ms total) 2024-01-21T03:27:10.014 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: 2024-01-21T03:27:10.014 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: [----------] Global test environment tear-down 2024-01-21T03:27:10.014 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (369129 ms total) 2024-01-21T03:27:10.014 INFO:tasks.workunit.client.0.smithi007.stdout: ec_list: [ PASSED ] 3 tests. 2024-01-21T03:27:23.002 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:37.333 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:23.007 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:37.338 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:23.066 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:37.397 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:23.067 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:37.343 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:23.068 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:37.399 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:23.071 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:37.347 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:23.073 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:37.405 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:23.119 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:37.395 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:23.139 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:37.470 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:43.075 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:57.406 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:43.079 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:57.410 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:43.121 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:57.452 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:43.124 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:57.400 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:43.127 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:57.403 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:43.128 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:57.458 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:43.131 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:57.462 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:43.170 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:57.446 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:43.189 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:57.521 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:27:45.835 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: Running main() from gmock_main.cc 2024-01-21T03:27:45.836 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-01-21T03:27:45.836 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [----------] Global test environment set-up. 2024-01-21T03:27:45.836 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-01-21T03:27:45.836 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-01-21T03:27:45.836 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (37106 ms) 2024-01-21T03:27:45.836 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-01-21T03:27:45.836 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (14965 ms) 2024-01-21T03:27:45.836 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-01-21T03:27:45.836 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (35778 ms) 2024-01-21T03:27:45.836 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-01-21T03:27:45.836 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (63852 ms) 2024-01-21T03:27:45.836 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-01-21T03:27:45.836 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (34902 ms) 2024-01-21T03:27:45.837 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-01-21T03:27:45.837 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (11806 ms) 2024-01-21T03:27:45.837 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-01-21T03:27:45.837 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (206496 ms) 2024-01-21T03:27:45.837 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (404906 ms total) 2024-01-21T03:27:45.837 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: 2024-01-21T03:27:45.837 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [----------] Global test environment tear-down 2024-01-21T03:27:45.837 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (404906 ms total) 2024-01-21T03:27:45.837 INFO:tasks.workunit.client.0.smithi007.stdout: write_operations: [ PASSED ] 7 tests. 2024-01-21T03:27:53.891 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-01-21T03:27:53.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-01-21T03:27:53.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-01-21T03:27:53.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-01-21T03:27:53.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-01-21T03:27:53.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-01-21T03:27:53.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-01-21T03:27:53.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: 2024-01-21T03:27:53.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-01-21T03:27:53.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-01-21T03:27:53.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (18381 ms) 2024-01-21T03:27:53.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-01-21T03:27:53.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (54025 ms) 2024-01-21T03:27:53.892 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (20078 ms) 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (14468 ms) 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (17065 ms) 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (35018 ms) 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (254150 ms) 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (413186 ms total) 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (413186 ms total) 2024-01-21T03:27:53.893 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-01-21T03:27:59.652 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: Running main() from gmock_main.cc 2024-01-21T03:27:59.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-01-21T03:27:59.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] Global test environment set-up. 2024-01-21T03:27:59.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-01-21T03:27:59.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-01-21T03:27:59.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.TooBig (6164 ms) 2024-01-21T03:27:59.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-01-21T03:27:59.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (321090 ms) 2024-01-21T03:27:59.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-01-21T03:27:59.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3691 ms) 2024-01-21T03:27:59.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-01-21T03:27:59.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (4357 ms) 2024-01-21T03:27:59.653 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3373 ms) 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3396 ms) 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3766 ms) 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3671 ms) 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (4358 ms) 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3830 ms) 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (4804 ms) 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3634 ms) 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-01-21T03:27:59.654 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3477 ms) 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (4492 ms) 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.Flush (3381 ms) 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3486 ms) 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3334 ms) 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3567 ms) 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3352 ms) 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3482 ms) 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3529 ms) 2024-01-21T03:27:59.655 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-01-21T03:27:59.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (4923 ms) 2024-01-21T03:27:59.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-01-21T03:27:59.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (4386 ms) 2024-01-21T03:27:59.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-01-21T03:27:59.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3707 ms) 2024-01-21T03:27:59.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-01-21T03:27:59.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (4360 ms) 2024-01-21T03:27:59.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-01-21T03:27:59.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3635 ms) 2024-01-21T03:27:59.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] 26 tests from LibRadosAio (419245 ms total) 2024-01-21T03:27:59.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: 2024-01-21T03:27:59.656 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-01-21T03:28:03.353 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:17.685 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:03.417 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:17.748 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:03.477 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:17.808 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:03.479 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:17.810 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:03.493 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:17.770 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:03.498 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:17.830 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:03.499 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:17.775 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:03.540 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:17.871 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:03.564 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:17.841 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:10.026 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:24.357 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:10.031 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:24.362 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:10.059 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:24.390 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:10.071 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:24.348 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:10.075 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:24.352 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:10.091 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:24.422 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:10.095 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:24.426 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:10.101 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:24.378 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:10.120 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:24.452 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:10.942 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrit watch_notify: Running main() from gmock_main.cc 2024-01-21T03:28:10.942 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-01-21T03:28:10.942 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: [----------] Global test environment set-up. 2024-01-21T03:28:10.942 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-01-21T03:28:10.942 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-01-21T03:28:10.942 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: handle_notify cookie 94567740418544 notify_id 1464583847938 notifier_gid 4624 2024-01-21T03:28:10.942 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (422179 ms) 2024-01-21T03:28:10.942 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-01-21T03:28:10.942 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: Trying... 2024-01-21T03:28:10.943 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: handle_notify cookie 94567753344128 notify_id 1486058684420 notifier_gid 22624 2024-01-21T03:28:10.943 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: Waiting for 3.000000000s 2024-01-21T03:28:10.943 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: Timed out. 2024-01-21T03:28:10.943 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: Flushing... 2024-01-21T03:28:10.943 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: Flushed... 2024-01-21T03:28:10.943 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7833 ms) 2024-01-21T03:28:10.943 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (430012 ms total) 2024-01-21T03:28:10.943 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: 2024-01-21T03:28:10.943 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: [----------] Global test environment tear-down 2024-01-21T03:28:10.943 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (430013 ms total) 2024-01-21T03:28:10.943 INFO:tasks.workunit.client.0.smithi007.stdout: watch_notify: [ PASSED ] 2 tests. 2024-01-21T03:28:23.249 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:37.580 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:23.254 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:37.585 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:23.294 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:37.625 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:23.307 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:37.583 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:23.310 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:37.586 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:23.312 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:37.644 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:23.314 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:37.646 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:23.342 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:37.618 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:23.350 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:37.681 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:24.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-01-21T03:28:24.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-01-21T03:28:24.969 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: seed 40460 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (201 ms) 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (113064 ms) 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (6510 ms) 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (2060 ms) 2024-01-21T03:28:24.970 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (234 ms) 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (84 ms) 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (95 ms) 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (220 ms) 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (97 ms) 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (393 ms) 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (203 ms) 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-01-21T03:28:24.971 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (202 ms) 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (322 ms) 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: osd_max_attr_size = 0 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (220775 ms) 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (10845 ms) 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: done waiting, doing copies 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: waiting for final deep scrubs... 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: done waiting 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (78175 ms) 2024-01-21T03:28:24.972 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-01-21T03:28:24.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (110 ms) 2024-01-21T03:28:24.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-01-21T03:28:24.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (42 ms) 2024-01-21T03:28:24.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-01-21T03:28:24.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4992 ms) 2024-01-21T03:28:24.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-01-21T03:28:24.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-01-21T03:28:24.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-01-21T03:28:24.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-01-21T03:28:24.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-01-21T03:28:24.973 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-01-21T03:28:43.507 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:57.838 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:43.511 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:57.843 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:43.579 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:57.856 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:43.582 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:57.914 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:43.585 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:57.861 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:43.588 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:57.919 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:43.612 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:57.943 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:43.647 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:57.924 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:43.666 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:57.997 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:28:54.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (438624 ms total) 2024-01-21T03:28:54.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-01-21T03:28:54.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-01-21T03:28:54.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-01-21T03:28:54.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (2685 ms) 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (2685 ms total) 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (1800 ms) 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (57 ms) 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (1857 ms total) 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (720 ms) 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (51 ms) 2024-01-21T03:28:54.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (772 ms total) 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (713 ms) 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (41 ms) 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (754 ms total) 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1640 ms) 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1640 ms total) 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-01-21T03:28:54.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (474154 ms total) 2024-01-21T03:28:54.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-01-21T03:29:03.767 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:18.098 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:03.771 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:18.103 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:03.848 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:18.180 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:03.851 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:18.183 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:03.854 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:18.131 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:03.858 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:18.135 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:03.900 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:18.232 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:03.916 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:18.193 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:03.931 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:18.262 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:23.931 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:38.262 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:23.963 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:38.240 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:23.966 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:38.243 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:23.974 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:38.306 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:24.028 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:38.359 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:24.047 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:38.324 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:24.085 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:38.416 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:24.087 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:38.419 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:24.147 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:38.478 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:44.035 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:58.367 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:44.036 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:58.312 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:44.040 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:58.372 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:44.044 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:58.322 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:44.099 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:58.376 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:44.113 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:58.445 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:44.124 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:58.455 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:44.125 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:58.457 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:44.196 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:58.527 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:56.575 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:10.906 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:56.579 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:10.911 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:56.624 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:10.956 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:56.637 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:10.969 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:56.641 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:10.973 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:56.647 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:10.924 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:56.648 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:10.925 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:56.707 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:11.039 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:29:56.709 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:10.986 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:00.749 INFO:tasks.workunit.client.0.smithi007.stdout:e 2024-01-21T03:30:00.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (8209 ms) 2024-01-21T03:30:00.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-01-21T03:30:00.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7712 ms) 2024-01-21T03:30:00.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-01-21T03:30:00.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7094 ms) 2024-01-21T03:30:00.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-01-21T03:30:00.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6904 ms) 2024-01-21T03:30:00.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-01-21T03:30:00.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (8643 ms) 2024-01-21T03:30:00.749 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (8462 ms) 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7201 ms) 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7298 ms) 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6926 ms) 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6822 ms) 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7237 ms) 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7111 ms) 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7987 ms) 2024-01-21T03:30:00.750 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-01-21T03:30:00.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6714 ms) 2024-01-21T03:30:00.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-01-21T03:30:00.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (8097 ms) 2024-01-21T03:30:00.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-01-21T03:30:00.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (8677 ms) 2024-01-21T03:30:00.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (121094 ms total) 2024-01-21T03:30:00.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: 2024-01-21T03:30:00.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] Global test environment tear-down 2024-01-21T03:30:00.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (540339 ms total) 2024-01-21T03:30:00.751 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ PASSED ] 42 tests. 2024-01-21T03:30:04.107 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:18.438 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:04.112 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:18.444 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:04.140 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:18.472 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:04.163 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:18.440 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:04.164 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:18.442 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:04.181 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:18.513 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:04.184 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:18.516 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:04.191 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:18.468 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:04.218 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:18.550 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:24.101 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:38.433 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:24.104 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:38.436 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:24.140 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:38.472 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:24.155 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:38.432 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:24.157 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:38.435 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:24.169 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:38.501 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:24.173 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:38.505 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:24.192 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:38.466 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:24.206 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:38.538 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:44.277 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:58.609 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:44.279 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:58.611 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:44.302 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:58.634 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:44.331 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:58.663 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:44.332 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:58.610 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:44.335 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:58.667 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:44.337 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:58.614 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:44.359 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:58.691 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:30:44.359 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:58.636 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:04.292 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:18.624 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:04.298 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:18.630 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:04.304 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:18.633 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:04.340 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:18.672 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:04.355 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:18.687 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:04.360 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:18.637 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:04.363 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:18.641 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:04.406 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:18.683 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:04.447 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:18.777 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:09.445 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:23.777 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:09.450 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:23.782 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:09.487 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:23.819 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:09.499 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:23.776 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:09.504 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:23.781 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:09.511 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:23.843 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:09.515 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:23.847 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:09.518 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:23.796 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:09.532 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:23.864 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:24.479 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:38.810 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:24.482 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:38.814 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:24.517 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:38.849 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:24.547 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:38.825 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:24.549 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:38.826 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:24.558 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:38.890 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:24.560 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:38.893 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:24.585 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:38.863 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:24.612 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:38.944 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:39.609 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:53.941 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:39.613 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:53.945 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:39.669 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:54.001 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:39.676 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:53.953 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:39.684 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:53.961 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:39.691 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:54.023 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:39.696 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:54.029 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:39.738 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:54.015 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:39.745 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:54.077 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:49.494 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:03.826 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:49.498 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:03.831 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:49.523 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:03.855 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:49.544 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:03.821 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:49.546 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:03.824 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:49.550 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:03.883 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:49.552 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:03.884 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:49.569 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:03.847 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:49.577 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:03.909 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:50.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-01-21T03:31:50.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-01-21T03:31:50.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-01-21T03:31:50.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: seed 40360 2024-01-21T03:31:50.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-01-21T03:31:50.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (47783 ms) 2024-01-21T03:31:50.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-01-21T03:31:50.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (52073 ms) 2024-01-21T03:31:50.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-01-21T03:31:50.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (4297 ms) 2024-01-21T03:31:50.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (104154 ms total) 2024-01-21T03:31:50.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 2024-01-21T03:31:50.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-01-21T03:31:50.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-01-21T03:31:50.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (75555 ms) 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (69852 ms) 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (26651 ms) 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: my_snaps [3] 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: my_snaps [4,3] 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: my_snaps [5,4,3] 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: promoting some heads 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: promoting from clones for snap 6 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: promoting from clones for snap 5 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: promoting from clones for snap 4 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: promoting from clones for snap 3 2024-01-21T03:31:50.337 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: waiting for scrubs... 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: done waiting 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (186988 ms) 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (12594 ms) 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (9666 ms) 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8202 ms) 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (11344 ms) 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (11471 ms) 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-01-21T03:31:50.338 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (12542 ms) 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (11839 ms) 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (42374 ms) 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (11016 ms) 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (11208 ms) 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (16730 ms) 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8732 ms) 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-01-21T03:31:50.339 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (11922 ms) 2024-01-21T03:31:54.511 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:08.844 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:54.516 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:08.848 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:54.542 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:08.875 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:54.562 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:08.840 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:54.563 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:08.841 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:54.568 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:08.900 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:54.570 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:08.903 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:54.578 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:08.856 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:31:54.586 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:08.918 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:09.710 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:24.042 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:09.712 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:24.044 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:09.749 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:24.081 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:09.762 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:24.094 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:09.763 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:24.095 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:09.769 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:24.046 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:09.770 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:24.047 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:09.802 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:24.134 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:09.806 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:24.084 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:13.185 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-01-21T03:32:13.185 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: hmm, no HitSet yet 2024-01-21T03:32:13.185 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: ok, hit_set contains 210:602f83fe:::foo:head 2024-01-21T03:32:13.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (10341 ms) 2024-01-21T03:32:13.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-01-21T03:32:13.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg_num = 32 2024-01-21T03:32:13.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 0 ls 1705807928,0 2024-01-21T03:32:13.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 1 ls 1705807928,0 2024-01-21T03:32:13.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 2 ls 1705807928,0 2024-01-21T03:32:13.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 3 ls 1705807928,0 2024-01-21T03:32:13.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 4 ls 1705807928,0 2024-01-21T03:32:13.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 5 ls 1705807928,0 2024-01-21T03:32:13.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 6 ls 1705807928,0 2024-01-21T03:32:13.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 7 ls 1705807928,0 2024-01-21T03:32:13.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 8 ls 1705807928,0 2024-01-21T03:32:13.186 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 9 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 10 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 11 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 12 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 13 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 14 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 15 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 16 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 17 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 18 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 19 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 20 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 21 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 22 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 23 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 24 ls 1705807928,0 2024-01-21T03:32:13.187 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 25 ls 1705807928,0 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 26 ls 1705807928,0 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 27 ls 1705807928,0 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 28 ls 1705807928,0 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 29 ls 1705807928,0 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 30 ls 1705807928,0 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg 31 ls 1705807928,0 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: pg_num = 32 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:6cac518f:::0:head 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:02547ec2:::1:head 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:f905c69b:::2:head 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:cfc208b3:::3:head 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:d83876eb:::4:head 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:b29083e3:::5:head 2024-01-21T03:32:13.188 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:c4fdafeb:::6:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:5c6b0b28:::7:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:bd63b0f1:::8:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:e960b815:::9:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:52ea6a34:::10:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:89d3ae78:::11:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:de5d7c5f:::12:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:566253c9:::13:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:62a1935d:::14:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:863748b0:::15:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:3958e169:::16:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:4d4dabf9:::17:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:8391935d:::18:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:28883081:::19:head 2024-01-21T03:32:13.189 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:69259c59:::20:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:4bdb80b7:::21:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:a11c5d71:::22:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:271af37b:::23:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:95b121be:::24:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:58d1031b:::25:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:0a050783:::26:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:c709704c:::27:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:cbe56eaf:::28:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:86b4b162:::29:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:70d89383:::30:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:dd450c7c:::31:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:6d5729b1:::32:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:c388f3fb:::33:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:56cfea31:::34:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:9dbc1bf7:::35:head 2024-01-21T03:32:13.190 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:40b74ccd:::36:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:4d5aaf42:::37:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:920f362c:::38:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:6cc53222:::39:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:9cad833f:::40:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:1ea84d41:::41:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:c4480ef6:::42:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:a694361e:::43:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:d1bd33e9:::44:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:ddc2cd5d:::45:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:2b782207:::46:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:7b187fca:::47:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:90ecdf6f:::48:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:a5ed95fe:::49:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:ea0eaa55:::50:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:f33ef17b:::51:head 2024-01-21T03:32:13.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:a0d1b2f6:::52:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:60c5229e:::53:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:edcbc575:::54:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:102cf253:::55:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:efb7fb0b:::56:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:50d0a326:::57:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:d4dc5daf:::58:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:3a130462:::59:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:ec87ed71:::60:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:d5bc9454:::61:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:3ddfe313:::62:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:7c2816b9:::63:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:47e00e4d:::64:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:c6410c18:::65:head 2024-01-21T03:32:13.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:b48ed237:::66:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:cd63ad31:::67:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:b179e92b:::68:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:0d9f741a:::69:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:6d3352ae:::70:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:c6d5c19e:::71:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:bc4729c3:::72:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:77e930b9:::73:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:0abeecfd:::74:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:b7c37e15:::75:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:b6378398:::76:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:02bd68de:::77:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:cc795d2d:::78:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:630d4fea:::79:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:e0d29ef5:::80:head 2024-01-21T03:32:13.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:fd6f13d2:::81:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:606461d5:::82:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:eadbdc43:::83:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:8761d0bb:::84:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:9ef0186f:::85:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:e0d41294:::86:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:961de695:::87:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:1423148f:::88:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:633a8fa2:::89:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:a8653809:::90:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:3dac8b33:::91:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:35aad435:::92:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:f6dcc343:::93:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:dbbdad87:::94:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:1cb48ce0:::95:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:03cd461c:::96:head 2024-01-21T03:32:13.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:17a4ea99:::97:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:9993c9a7:::98:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:6394211c:::99:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:94c7ae57:::100:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:6fdee5bb:::101:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:9a477fd1:::102:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:eb850916:::103:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:affc56b9:::104:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:b42dc814:::105:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:f319f8f0:::106:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:9a40b9de:::107:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:8b524f28:::108:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:e3de589f:::109:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:90f90a5b:::110:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:a7b4f1d7:::111:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:af51766e:::112:head 2024-01-21T03:32:13.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:b6f90bd1:::113:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:e0261208:::114:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:c9569ef7:::115:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:61bebe50:::116:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:fe93412b:::117:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:d3d38bee:::118:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:3100ba0c:::119:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:d0560ada:::120:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:f0ea8b35:::121:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:766f231a:::122:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:a07a2582:::123:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:bd7c6b3a:::124:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:fb2ddaff:::125:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:4408e1fe:::126:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:ee1df7a7:::127:head 2024-01-21T03:32:13.196 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:c3002909:::128:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:4f48ffa9:::129:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:edf38733:::130:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:c08425c0:::131:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:5f902d98:::132:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:41ea2c93:::133:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:813cee13:::134:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:0131818d:::135:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:26ba5a85:::136:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:381b8a5a:::137:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:28797e47:::138:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:bfca7f22:::139:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:36807075:::140:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:80b03975:::141:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:5c15709b:::142:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:f39ea15e:::143:head 2024-01-21T03:32:13.197 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:ea992956:::144:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:48887b1c:::145:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:9f24a9dd:::146:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:987f100b:::147:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:d2dd3581:::148:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:7fed1808:::149:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:c80b70e9:::150:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:85ed90f9:::151:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:36428b24:::152:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:d044c34a:::153:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:7c18bf58:::154:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:d1c21232:::155:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:a7a3c575:::156:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:87da0633:::157:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:d5ac3822:::158:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:3f20522d:::159:head 2024-01-21T03:32:13.198 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:6ca26563:::160:head 2024-01-21T03:32:13.199 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:532ce135:::161:head 2024-01-21T03:32:13.199 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:c78863e6:::162:head 2024-01-21T03:32:24.778 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:39.110 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:24.783 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:39.116 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:24.820 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:39.152 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:24.833 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:39.111 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:24.841 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:39.118 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:24.874 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:39.152 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:24.906 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:39.238 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:24.914 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:39.246 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:24.948 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:39.280 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:29.661 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:43.993 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:29.664 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:43.997 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:29.683 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:44.015 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:29.712 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:43.989 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:29.714 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:43.991 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:29.719 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:44.052 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:29.721 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:44.053 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:29.729 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:44.007 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:29.738 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:44.071 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:39.674 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:54.007 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:39.678 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:54.011 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:39.697 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:54.029 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:39.726 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:54.004 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:39.729 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:54.061 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:39.730 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:54.007 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:39.732 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:54.064 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:39.743 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:54.021 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:39.748 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:54.080 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:44.963 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:59.295 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:44.972 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:59.304 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:45.004 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:59.336 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:45.037 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:59.315 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:45.040 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:59.318 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:45.054 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:59.386 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:45.057 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:59.390 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:45.066 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:59.344 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:45.091 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:59.423 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:50.149 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:04.482 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:50.154 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:04.487 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:50.173 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:04.506 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:50.201 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:04.479 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:50.203 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:04.481 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:50.218 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:04.496 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:50.233 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:04.565 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:50.234 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:04.567 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:50.250 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:04.582 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:55.015 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:09.347 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:55.019 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:09.352 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:55.036 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:09.368 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:55.064 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:09.342 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:55.066 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:09.343 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:55.068 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:09.401 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:55.071 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:09.404 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:55.083 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:09.361 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:32:55.090 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:09.423 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:10.311 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:24.644 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:10.314 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:24.647 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:10.359 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:24.691 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:10.359 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:24.637 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:10.360 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:24.638 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:10.375 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:24.708 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:10.377 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:24.710 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:10.392 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:24.669 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:10.420 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:24.753 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:24.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:7462ddf6:::163:head 2024-01-21T03:33:24.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:5d165639:::164:head 2024-01-21T03:33:24.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:f43765fc:::165:head 2024-01-21T03:33:24.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:b4c720e9:::166:head 2024-01-21T03:33:24.824 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:e694b040:::167:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:afa38db2:::168:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:77ba9f53:::169:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:87495034:::170:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:7c96bf0e:::171:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:dbe346cc:::172:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:e943ec24:::173:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:f97a9c0c:::174:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:6f26e74d:::175:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:4f95e106:::176:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:0e6f2f8f:::177:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:05db05f1:::178:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:38a78d66:::179:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:d095610b:::180:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:a1a9d709:::181:head 2024-01-21T03:33:24.825 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:1e5d39db:::182:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:f7df4fb9:::183:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:03a7f161:::184:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:ba70721e:::185:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:28e5662d:::186:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:973d52de:::187:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:4303eb1c:::188:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:b990b48e:::189:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:29b8165b:::190:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:3547f197:::191:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:7e260936:::192:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:1abec7b1:::193:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:10fdda93:::194:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:15817eea:::195:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:770bab57:::196:head 2024-01-21T03:33:24.826 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:ed9e13e7:::197:head 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:71471a8f:::198:head 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: checking for 211:10fb1d02:::199:head 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (21156 ms) 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705807950,0 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: first is 1705807950 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705807950,0 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705807950,0 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705807950,1705807952,1705807953,0 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705807950,1705807952,1705807953,0 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705807950,1705807952,1705807953,0 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705807950,1705807952,1705807953,1705807955,1705807956,0 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705807950,1705807952,1705807953,1705807955,1705807956,0 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705807950,1705807952,1705807953,1705807955,1705807956,0 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705807950,1705807952,1705807953,1705807955,1705807956,1705807958,1705807959,0 2024-01-21T03:33:24.827 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705807950,1705807952,1705807953,1705807955,1705807956,1705807958,1705807959,0 2024-01-21T03:33:24.828 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705807950,1705807952,1705807953,1705807955,1705807956,1705807958,1705807959,0 2024-01-21T03:33:24.828 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705807953,1705807955,1705807956,1705807958,1705807959,1705807961,1705807962,0 2024-01-21T03:33:24.828 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: first now 1705807953, trimmed 2024-01-21T03:33:24.828 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22357 ms) 2024-01-21T03:33:24.828 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-01-21T03:33:24.828 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: foo0 2024-01-21T03:33:24.828 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-01-21T03:33:24.828 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (18996 ms) 2024-01-21T03:33:24.828 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-01-21T03:33:30.211 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:44.544 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:30.214 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:44.547 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:30.262 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:44.594 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:30.267 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:44.599 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:30.271 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:44.603 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:30.271 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:44.549 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:30.272 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:44.550 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:30.302 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:44.635 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:30.304 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:44.582 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:30.950 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:45.227 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:30.951 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:45.229 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:30.960 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:45.237 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:31.024 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:45.357 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:31.026 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:45.358 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:31.037 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:45.370 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:31.038 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:45.371 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:31.038 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:45.371 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:31.048 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:45.381 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:45.293 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:59.625 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:45.296 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:59.629 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:45.334 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:59.667 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:45.343 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:59.621 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:45.346 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:59.624 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:45.357 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:59.690 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:45.362 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:59.695 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:45.369 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:59.647 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:45.388 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:59.721 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:50.459 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:04.792 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:50.463 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:04.796 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:50.488 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:04.821 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:50.516 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:04.849 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:50.518 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:04.795 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:50.520 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:04.853 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:50.523 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:04.802 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:50.553 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:04.831 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:33:50.570 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:04.902 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:00.363 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:14.695 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:00.364 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:14.697 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:00.401 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:14.734 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:00.417 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:14.695 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:00.418 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:14.750 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:00.419 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:14.752 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:00.420 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:14.698 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:00.442 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:14.720 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:00.451 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:14.784 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:14.256 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:28.589 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:14.274 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:28.607 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:14.324 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:28.657 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:14.325 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:28.603 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:14.332 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:28.664 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:14.342 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:28.620 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:14.344 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:28.677 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:14.398 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:28.676 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:14.400 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:28.733 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:15.451 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:29.783 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:15.499 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:29.777 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:15.500 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:29.779 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:15.513 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:29.846 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:15.516 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:29.794 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:15.537 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:29.870 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:15.587 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:29.920 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:15.591 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:29.923 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:15.609 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:29.942 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:16.088 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:30.366 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:16.089 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:30.367 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:16.099 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:30.377 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:16.250 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:30.583 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:16.250 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:30.583 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:16.251 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:30.584 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:16.251 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:30.584 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:16.259 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:30.592 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:16.260 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:30.592 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:30.532 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:44.864 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:30.532 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:44.865 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:30.534 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:44.867 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:30.558 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:44.891 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:30.578 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:44.854 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:30.581 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:44.859 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:30.599 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:44.932 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:30.604 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:44.882 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:30.624 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:44.957 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:35.679 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:50.012 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:35.685 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:50.018 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:35.709 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:50.042 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:35.737 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:50.015 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:35.740 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:50.018 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:35.740 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:50.073 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:35.746 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:50.079 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:35.753 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:50.031 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:35.766 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:50.098 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:50.590 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:04.922 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:50.592 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:04.925 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:50.613 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:04.946 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:50.640 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:04.918 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:50.642 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:04.920 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:50.662 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:04.940 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:50.664 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:04.997 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:50.666 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:04.999 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:50.685 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:05.018 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:55.811 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:10.143 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:55.816 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:10.149 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:55.836 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:10.169 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:55.865 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:10.143 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:55.870 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:10.148 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:55.887 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:10.165 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:55.888 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:10.221 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:55.894 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:10.227 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:34:55.919 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:10.252 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:05.762 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:20.095 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:05.813 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:20.091 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:05.814 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:20.092 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:05.818 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:20.151 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:05.841 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:20.119 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:05.843 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:20.176 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:05.873 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:20.206 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:05.876 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:20.209 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:05.899 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:20.232 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:15.838 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:30.171 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:15.843 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:30.176 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:15.870 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:30.203 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:15.895 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:30.173 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:15.896 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:30.229 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:15.899 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:30.232 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:15.901 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:30.179 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:15.924 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:30.257 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:15.928 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:30.207 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:20.937 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:35.270 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:20.989 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:35.268 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:20.991 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:35.270 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:20.999 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:35.332 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:21.003 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:35.336 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:21.003 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:35.336 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:21.008 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:35.286 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:21.025 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:35.358 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:21.029 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:35.362 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:35.805 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:50.137 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:35.808 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:50.142 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:35.848 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:50.181 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:35.853 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:50.131 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:35.854 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:50.132 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:35.870 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:50.203 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:35.873 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:50.207 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:35.876 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:50.155 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:35.899 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:50.232 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:36.438 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:50.716 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:36.439 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:50.717 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:36.447 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:50.725 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:36.516 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:50.848 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:36.516 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:50.850 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:36.525 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:50.858 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:36.535 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:50.868 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:36.536 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:50.869 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:36.544 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:50.877 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:56.078 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:10.410 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:56.082 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:10.414 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:56.119 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:10.451 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:56.136 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:10.465 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:56.136 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:10.466 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:56.136 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:10.413 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:56.138 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:10.416 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:56.160 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:10.492 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:35:56.162 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:10.440 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:16.064 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:30.396 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:16.069 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:30.346 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:16.069 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:30.401 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:16.073 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:30.405 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:16.073 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:30.350 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:16.076 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:30.408 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:16.186 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:30.518 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:16.187 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:30.518 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:16.254 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:30.531 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:31.032 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:45.363 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:31.035 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:45.366 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:31.067 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:45.399 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:31.085 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:45.361 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:31.088 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:45.419 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:31.089 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:45.366 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:31.090 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:45.422 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:31.118 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:45.449 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:31.118 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:45.395 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:35.992 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:50.323 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:36.037 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:50.314 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:36.042 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:50.318 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:36.042 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:50.374 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:36.055 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:50.331 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:36.057 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:50.388 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:36.104 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:50.435 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:36.108 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:50.439 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:36.120 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:50.451 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:43.738 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:58.069 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:43.795 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:58.125 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:43.811 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:17:58.141 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:43.820 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:36:43.830+0000 25e84640 -1 snap_mapper.add_oid found existing snaps mapped on 229:efab6a75:test-rados-api-smithi007-40360-80::foo:2, removing 2024-01-21T03:36:43.835 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:36:43.840+0000 2ca8f640 -1 snap_mapper.add_oid found existing snaps mapped on 229:efab6a75:test-rados-api-smithi007-40360-80::foo:2, removing 2024-01-21T03:36:43.846 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:58.177 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:43.847 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:58.178 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:43.857 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:58.133 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:43.861 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:58.137 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:43.864 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:58.195 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:43.876 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:58.152 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:51.112 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:05.442 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:51.116 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:05.447 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:51.140 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:05.470 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:51.160 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:05.436 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:51.163 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:05.439 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:51.170 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:05.500 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:51.172 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:05.502 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:51.185 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:05.460 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:36:51.245 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:05.576 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:01.106 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:15.436 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:01.152 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:15.427 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:01.155 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:15.429 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:01.157 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:15.487 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:01.180 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:15.456 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:01.185 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:15.516 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:01.211 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:15.541 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:01.212 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:15.542 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:01.249 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:15.580 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:06.167 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:20.497 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:06.171 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:20.501 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:06.194 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:20.525 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:06.217 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:20.493 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:06.220 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:20.496 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:06.221 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:20.552 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:06.229 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:20.559 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:06.238 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:20.514 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:06.245 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:20.575 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:19.679 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (21639 ms) 2024-01-21T03:37:19.680 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-01-21T03:37:19.680 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (25300 ms) 2024-01-21T03:37:19.680 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-01-21T03:37:19.680 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3650 ms) 2024-01-21T03:37:19.680 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-01-21T03:37:19.680 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.680 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (4533 ms) 2024-01-21T03:37:19.680 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-01-21T03:37:19.680 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (5678 ms) 2024-01-21T03:37:19.680 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4401 ms) 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5699 ms) 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40887 ms) 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17915 ms) 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5005 ms) 2024-01-21T03:37:19.681 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (5053 ms) 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (27261 ms) 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16751 ms) 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6785 ms) 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5717 ms) 2024-01-21T03:37:19.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-01-21T03:37:19.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: waiting for scrubs... 2024-01-21T03:37:19.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: done waiting 2024-01-21T03:37:19.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25482 ms) 2024-01-21T03:37:19.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-01-21T03:37:19.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12354 ms) 2024-01-21T03:37:19.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-01-21T03:37:19.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11438 ms) 2024-01-21T03:37:19.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-01-21T03:37:19.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:37:19.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (10944 ms) 2024-01-21T03:37:21.187 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:35.513 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:21.234 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:35.509 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:21.235 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:35.510 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:21.253 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:35.583 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:21.269 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:35.544 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:21.299 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:35.629 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:21.343 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:35.672 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:21.348 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:35.678 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:21.387 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:35.716 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:21.979 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:36.254 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:21.980 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:36.255 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:21.988 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:36.263 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:22.379 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:36.708 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:22.380 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:36.710 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:22.397 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:36.727 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:22.475 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:36.804 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:22.481 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:36.810 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:22.498 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:36.827 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:31.352 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:45.681 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:31.355 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:45.684 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:31.399 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:45.729 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:31.413 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:45.687 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:31.414 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:45.689 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:31.414 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:45.744 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:31.416 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:45.746 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:31.446 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:45.721 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:31.451 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:45.780 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:41.397 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:55.725 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:41.398 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:55.727 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:41.423 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:55.752 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:41.454 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:55.728 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:41.456 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:55.730 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:41.457 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:55.786 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:41.463 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:55.792 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:41.482 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:55.757 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:41.496 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:55.825 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:42.214 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:56.488 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:42.215 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:56.489 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:42.227 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:56.501 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:42.294 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:56.623 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:42.295 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:56.624 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:42.305 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:18:56.634 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:42.356 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:56.685 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:42.357 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:56.686 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:42.366 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:18:56.695 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:56.458 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:10.786 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:56.458 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:10.786 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:56.460 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:10.789 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:56.486 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:10.814 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:56.511 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:10.839 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:56.514 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:10.787 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:56.516 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:10.789 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:56.539 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:10.865 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:37:56.543 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:10.817 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:01.567 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:15.896 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:01.569 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:15.898 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:01.581 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:15.910 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:01.624 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:15.898 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:01.627 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:15.956 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:01.629 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:15.902 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:01.630 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:15.958 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:01.641 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:15.914 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:01.648 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:15.976 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:11.653 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:25.981 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:11.656 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:25.984 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:11.676 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:26.004 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:11.699 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:25.973 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:11.703 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:25.976 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:11.714 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:26.042 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:11.717 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:26.045 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:11.727 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:26.000 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:11.738 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:26.066 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:21.647 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:35.974 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:21.648 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:35.976 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:21.668 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:35.996 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:21.692 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:35.965 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:21.694 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:35.967 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:21.699 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:36.027 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:21.701 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:36.029 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:21.712 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:35.985 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:21.722 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:36.050 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:26.563 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:40.891 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:26.568 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:40.896 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:26.581 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:40.909 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:26.615 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:40.888 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:26.618 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:40.891 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:26.619 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:40.947 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:26.620 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:40.948 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:26.629 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:40.902 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:26.631 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:40.959 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:27.843 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T03:38:27.847+0000 25e84640 -1 snap_mapper.add_oid found existing snaps mapped on 91:159b1ae7:test-rados-api-smithi007-40360-97::foo:21, removing 2024-01-21T03:38:27.866 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T03:38:27.873+0000 2d290640 -1 snap_mapper.add_oid found existing snaps mapped on 91:159b1ae7:test-rados-api-smithi007-40360-97::foo:21, removing 2024-01-21T03:38:41.625 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:55.952 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:41.627 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:55.954 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:41.657 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:55.984 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:41.667 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:55.940 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:41.669 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:55.942 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:41.674 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:56.002 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:41.677 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:56.004 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:41.703 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:55.976 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:41.709 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:56.036 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:42.433 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:56.705 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:42.434 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:56.706 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:42.443 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:56.716 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:42.585 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:56.913 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:42.587 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:56.914 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:42.587 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:56.915 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:42.588 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:56.916 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:42.597 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:19:56.924 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:42.598 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:19:56.926 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:49.987 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:04.314 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:49.993 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:04.320 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:50.023 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:04.350 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:50.042 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:04.315 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:50.046 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:04.318 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:50.050 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:04.378 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:50.057 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:04.385 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:50.069 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:04.341 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:50.083 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:04.410 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:56.610 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:10.937 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:56.661 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:10.988 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:56.685 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:11.011 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:56.709 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:10.981 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:56.712 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:10.984 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:56.715 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:11.042 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:56.717 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:11.044 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:56.732 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:11.004 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:38:56.738 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:11.065 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:01.895 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:16.222 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:01.902 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:16.229 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:01.929 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:16.255 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:01.960 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:16.233 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:01.967 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:16.239 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:01.978 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:16.305 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:01.986 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:16.313 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:01.993 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:16.265 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:02.014 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:16.340 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:11.737 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:26.064 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:11.739 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:26.066 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:11.765 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:26.092 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:11.790 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:26.116 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:11.792 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:26.064 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:11.793 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:26.065 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:11.794 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:26.120 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:11.817 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:26.088 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:11.819 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:26.146 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:21.834 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:36.160 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:21.835 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:36.162 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:21.862 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:36.188 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:21.877 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:36.204 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:21.881 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:36.152 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:21.882 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:36.154 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:21.883 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:36.210 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:21.910 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:36.181 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:21.910 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:36.236 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:26.976 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:41.303 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:27.028 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:41.354 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:27.041 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:41.312 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:27.043 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:41.314 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:27.050 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:41.376 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:27.062 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:41.333 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:27.089 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:41.415 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:27.090 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:41.416 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:27.107 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:41.433 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:41.918 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:56.243 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:41.980 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:56.305 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.017 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:56.343 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.033 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:56.304 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.036 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:56.307 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.049 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:56.375 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.051 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:56.377 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.069 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:56.340 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.094 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:56.420 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.727 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:56.998 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.729 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:57.000 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.740 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:57.011 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.867 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:57.193 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.868 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:57.194 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.877 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:20:57.203 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.900 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:57.226 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.902 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:57.227 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:42.913 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:20:57.239 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:57.021 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:11.291 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:57.023 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:11.294 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:57.024 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:11.349 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:57.025 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:11.351 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:57.061 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:11.387 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:57.064 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:11.335 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:57.081 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:11.406 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:57.083 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:11.409 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:39:57.117 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:11.443 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:02.061 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:16.387 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:02.066 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:16.391 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:02.091 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:16.416 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:02.110 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:16.380 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:02.112 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:16.383 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:02.123 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:16.448 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:02.124 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:16.450 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:02.127 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:16.398 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:02.140 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:16.466 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:12.091 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:26.416 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:12.095 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:26.420 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:12.124 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:26.449 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:12.142 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:26.412 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:12.149 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:26.419 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:12.150 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:26.475 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:12.160 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:26.485 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:12.172 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:26.442 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:12.187 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:26.512 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:22.069 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:36.393 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:22.122 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:36.446 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:22.123 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:36.448 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:22.125 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:36.450 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:22.150 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:36.475 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:22.151 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:36.476 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:22.177 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:36.447 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:22.180 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:36.451 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:22.201 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:36.471 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:32.254 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:46.579 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:32.260 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:46.530 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:32.261 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:46.586 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:32.265 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:46.535 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:32.284 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:46.554 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:32.289 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:46.614 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:32.334 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:46.659 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:32.336 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:46.661 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:32.351 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:46.676 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:42.315 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:56.639 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:42.367 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:56.692 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:42.369 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:56.638 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:42.372 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:56.642 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:42.397 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:56.667 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:42.405 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:21:56.730 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:42.425 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:56.750 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:42.433 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:56.758 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:42.462 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:21:56.786 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:47.224 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:01.549 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:47.228 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:01.552 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:47.245 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:01.570 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:47.276 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:01.600 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:47.278 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:01.602 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:47.282 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:01.552 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:47.284 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:01.553 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:47.294 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:01.618 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:47.353 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:01.623 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:48.041 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:02.310 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:48.042 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:02.311 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:48.054 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:02.323 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:48.121 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:02.445 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:48.122 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:02.447 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:48.134 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:02.459 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:48.136 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:02.460 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:48.136 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:02.461 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:40:48.155 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:02.479 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:02.453 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:16.776 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:02.454 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:16.778 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:02.490 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:16.814 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:02.501 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:16.771 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:02.505 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:16.774 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:02.511 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:16.835 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:02.516 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:16.840 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:02.548 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:16.817 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:02.564 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:16.888 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:17.381 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:31.705 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:17.382 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:31.706 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:17.418 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:31.742 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:17.428 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:31.697 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:17.432 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:31.755 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:17.433 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:31.702 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:17.434 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:31.758 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:17.467 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:31.736 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:17.474 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:31.798 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:22.555 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:36.879 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:22.615 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:36.939 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:22.643 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:36.966 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:22.677 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:37.001 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:22.683 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:37.006 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:22.684 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:36.953 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:22.686 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:36.955 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:22.705 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:37.029 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:22.707 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:36.976 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:28.981 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-01-21T03:41:28.982 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:41:28.982 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (10915 ms) 2024-01-21T03:41:28.982 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-01-21T03:41:28.982 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:41:28.982 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (7985 ms) 2024-01-21T03:41:28.982 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-01-21T03:41:28.982 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:41:28.982 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6831 ms) 2024-01-21T03:41:28.982 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-01-21T03:41:28.982 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:41:28.982 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (28365 ms) 2024-01-21T03:41:28.982 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-01-21T03:41:28.982 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15417 ms) 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (15269 ms) 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 00000010 69 73 |is| 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 00000012 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (15185 ms) 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7102 ms) 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (975097 ms total) 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-01-21T03:41:28.983 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1824 ms) 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (12545 ms) 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18169 ms) 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (16 ms) 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (32554 ms total) 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7905 ms) 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (9944 ms) 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-01-21T03:41:28.984 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: waiting for scrub... 2024-01-21T03:41:28.985 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: done waiting 2024-01-21T03:41:28.985 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (28189 ms) 2024-01-21T03:41:28.985 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-01-21T03:41:28.985 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (12259 ms) 2024-01-21T03:41:28.985 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-01-21T03:41:28.985 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8704 ms) 2024-01-21T03:41:28.985 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-01-21T03:41:28.985 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (9333 ms) 2024-01-21T03:41:28.985 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-01-21T03:41:28.985 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (12300 ms) 2024-01-21T03:41:28.985 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-01-21T03:41:32.629 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:46.952 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:32.634 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:46.958 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:32.661 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:22:46.985 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:32.686 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:46.955 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:32.692 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:46.961 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:32.710 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:47.034 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:32.713 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:47.036 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:32.727 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:46.996 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:32.746 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:22:47.070 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:47.565 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:01.888 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:47.567 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:01.891 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:47.596 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:01.919 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:47.613 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:01.881 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:47.615 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:01.883 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:47.618 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:01.941 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:47.621 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:01.945 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:47.641 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:01.909 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:47.649 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:01.972 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:48.458 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:02.727 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:48.460 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:02.728 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:48.469 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:02.737 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:48.561 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:02.884 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:48.563 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:02.886 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:48.576 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:02.899 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:48.581 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:02.904 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:48.582 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:02.906 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:41:48.592 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:02.915 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:02.681 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:17.003 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:02.685 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:17.008 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:02.718 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:17.041 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:02.736 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:17.059 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:02.737 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:17.005 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:02.739 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:17.007 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:02.741 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:17.064 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:02.770 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:17.038 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:02.776 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:17.099 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:07.651 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:21.973 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:07.697 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:21.965 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:07.698 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:21.966 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:07.701 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:22.024 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:07.715 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:21.983 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:07.722 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:22.045 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:07.753 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:22.076 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:07.754 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:22.077 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:07.771 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:22.094 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:17.751 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:32.073 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:17.753 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:32.076 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:17.789 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:32.112 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:17.803 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:32.070 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:17.805 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:32.128 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:17.806 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:32.129 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:17.811 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:32.079 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:17.836 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:32.159 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:17.843 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:32.111 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:27.917 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:42.239 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:27.971 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:42.294 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:27.983 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:42.251 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:27.986 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:42.254 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:27.997 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:42.320 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:28.014 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:42.281 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:28.021 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:42.344 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:28.022 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:42.345 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:28.047 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:42.370 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:32.964 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:47.286 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:32.969 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:47.292 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:32.975 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:47.242 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:32.979 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:47.247 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:32.996 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:47.318 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:33.001 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:47.269 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:33.028 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:47.350 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:33.035 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:47.357 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:33.059 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:47.381 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:44.995 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:59.317 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:45.007 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:59.329 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:45.013 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:59.280 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:45.022 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:59.289 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:45.023 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:59.345 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:45.033 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:59.355 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:45.051 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:23:59.373 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:45.106 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:59.374 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:45.125 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:23:59.447 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:47.867 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:02.189 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:47.870 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:02.192 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:47.884 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:02.207 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:47.912 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:02.179 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:47.916 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:02.184 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:47.919 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:02.241 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:47.922 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:02.244 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:47.928 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:02.196 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:47.934 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:02.256 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:48.659 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:02.926 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:48.660 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:02.927 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:48.669 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:02.936 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:48.810 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:03.132 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:48.811 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:03.133 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:48.811 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:03.133 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:48.812 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:03.134 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:48.822 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:03.144 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:42:48.822 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:03.144 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:07.960 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:22.281 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.019 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:22.286 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.023 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:22.290 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.027 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:22.349 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.068 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:22.335 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.087 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:22.409 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.090 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:22.412 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.109 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:22.431 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.122 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:22.444 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.756 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:23.023 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.757 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:23.024 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.764 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:23.031 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.846 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:23.168 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.847 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:23.169 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.861 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:23.183 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.861 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:23.183 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.863 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:23.185 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:08.874 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:23.196 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:23.032 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:37.352 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:23.036 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:37.357 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:23.075 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:37.397 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:23.083 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:37.350 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:23.087 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:37.354 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:23.087 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:37.409 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:23.091 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:37.412 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:23.123 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:37.390 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:23.124 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:37.446 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:28.066 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:42.388 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:28.068 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:42.390 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:28.083 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:42.405 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:28.111 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:42.378 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:28.112 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:42.379 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:28.117 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:42.438 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:28.118 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:42.440 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:28.127 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:42.394 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:28.134 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:42.456 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:38.105 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:52.427 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:38.109 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:52.431 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:38.126 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:24:52.447 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:38.143 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:52.410 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:38.144 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:52.465 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:38.144 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:52.411 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:38.146 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:52.467 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:38.161 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:52.427 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:38.161 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:24:52.483 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:48.143 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:02.464 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:48.161 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:02.482 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:48.188 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:02.509 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:48.192 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:02.459 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:48.194 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:02.461 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:48.203 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:02.524 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:48.206 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:02.528 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:48.217 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:02.483 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:48.231 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:02.552 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:53.297 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:07.617 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:53.302 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:07.623 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:53.304 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:07.571 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:53.307 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:07.628 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:53.308 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:07.574 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:53.309 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:07.630 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:53.387 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:07.653 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:53.398 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:07.719 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:43:53.400 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:07.721 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:08.410 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:22.731 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:08.414 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:22.735 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:08.458 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:22.779 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:08.469 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:22.735 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:08.479 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:22.745 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:08.485 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:22.806 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:08.492 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:22.813 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:08.508 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:22.774 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:08.529 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:22.850 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:23.460 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:37.780 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:23.533 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:37.799 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:23.535 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:37.801 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:23.535 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:37.856 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:23.574 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:37.894 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:23.581 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:37.901 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:23.583 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:37.849 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:23.583 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:37.904 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:23.639 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:37.960 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:28.391 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:42.711 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:28.467 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:42.788 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:28.471 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:42.791 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:28.506 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:42.826 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:28.520 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:42.786 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:28.540 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:42.861 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:28.545 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:42.811 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:28.564 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:42.830 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:28.565 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:42.886 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:33.562 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:47.882 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:33.570 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:47.891 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:33.594 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:47.914 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:33.623 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:47.888 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:33.626 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:47.892 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:33.647 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:47.967 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:33.647 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:47.913 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:33.648 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:47.969 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:33.671 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:47.991 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:34.551 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (10034 ms) 2024-01-21T03:44:34.552 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-01-21T03:44:34.552 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14607 ms) 2024-01-21T03:44:34.552 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-01-21T03:44:34.552 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (11103 ms) 2024-01-21T03:44:34.552 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-01-21T03:44:34.552 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15190 ms) 2024-01-21T03:44:34.552 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-01-21T03:44:34.552 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (10084 ms) 2024-01-21T03:44:34.552 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-01-21T03:44:34.552 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (11077 ms) 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: hmm, no HitSet yet 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: ok, hit_set contains 261:602f83fe:::foo:head 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9419 ms) 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705808569,0 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: first is 1705808569 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705808569,0 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705808569,0 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705808569,1705808572,1705808573,0 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705808569,1705808572,1705808573,0 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705808569,1705808572,1705808573,0 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705808569,1705808572,1705808573,1705808575,1705808576,0 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705808569,1705808572,1705808573,1705808575,1705808576,0 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705808569,1705808572,1705808573,1705808575,1705808576,0 2024-01-21T03:44:34.553 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705808569,1705808572,1705808573,1705808575,1705808576,1705808578,1705808579,0 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705808569,1705808572,1705808573,1705808575,1705808576,1705808578,1705808579,0 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705808569,1705808572,1705808573,1705808575,1705808576,1705808578,1705808579,0 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: got ls 1705808573,1705808575,1705808576,1705808578,1705808579,1705808581,1705808582,0 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: first now 1705808573, trimmed 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (24022 ms) 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: foo0 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (18220 ms) 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (21808 ms) 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (25357 ms) 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-01-21T03:44:34.554 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (6881 ms) 2024-01-21T03:44:34.555 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-01-21T03:44:34.555 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4542 ms) 2024-01-21T03:44:34.555 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-01-21T03:44:34.555 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4487 ms) 2024-01-21T03:44:34.555 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-01-21T03:44:34.555 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (5391 ms) 2024-01-21T03:44:34.555 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (280858 ms total) 2024-01-21T03:44:34.555 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: 2024-01-21T03:44:34.555 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-01-21T03:44:34.555 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1414032 ms total) 2024-01-21T03:44:34.555 INFO:tasks.workunit.client.0.smithi007.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-01-21T03:44:34.557 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:44:34.557 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40149 2024-01-21T03:44:34.557 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40149 2024-01-21T03:44:34.557 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:44:34.557 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40341 2024-01-21T03:44:34.557 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40341 2024-01-21T03:44:34.557 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:44:34.557 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40846 2024-01-21T03:44:34.557 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40846 2024-01-21T03:44:34.557 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:44:34.557 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40579 2024-01-21T03:44:34.557 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40579 2024-01-21T03:44:34.557 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40434 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40434 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40222 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40222 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40522 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40522 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40987 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40987 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41006 2024-01-21T03:44:34.558 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 41006 2024-01-21T03:44:38.717 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:53.037 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:38.722 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:53.042 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:38.735 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:25:53.055 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:38.769 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:53.089 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:38.771 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:53.091 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:38.774 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:53.040 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:38.776 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:53.042 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:38.786 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:25:53.107 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:38.790 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:53.056 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:48.600 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:02.920 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:48.604 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:02.924 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:48.617 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:02.937 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:48.654 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:02.919 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:48.654 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:02.974 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:48.656 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:02.976 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:48.657 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:02.922 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:48.670 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:02.935 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:48.674 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:02.994 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:53.645 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:07.965 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:53.651 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:07.971 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:53.665 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:07.986 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:53.699 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:07.965 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:53.703 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:07.968 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:53.705 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:08.025 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:53.712 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:08.032 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:53.716 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:07.982 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:53.729 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:08.049 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:58.658 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:12.978 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:58.660 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:12.980 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:58.673 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:12.993 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:58.708 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:12.973 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:58.709 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:13.030 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:58.710 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:12.976 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:58.716 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:13.036 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:58.722 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:12.987 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:44:58.729 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:13.049 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:03.821 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:18.141 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:03.826 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:18.146 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:03.844 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:18.164 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:03.872 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:18.137 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:03.876 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:18.142 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:03.891 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:18.156 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:03.903 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:18.223 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:03.905 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:18.225 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:03.922 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:18.242 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:08.692 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:23.012 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:08.694 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:23.014 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:08.704 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:23.025 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:08.744 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:23.065 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:08.745 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:23.010 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:08.746 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:23.012 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:08.747 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:23.066 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:08.757 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:23.022 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:08.758 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:23.078 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:18.723 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:33.043 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:18.727 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:33.047 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:18.743 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:33.063 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:18.773 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:33.037 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:18.775 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:33.040 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:18.776 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:33.096 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:18.778 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:33.098 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:18.787 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:33.053 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:18.794 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:33.114 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:28.769 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:43.089 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:28.771 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:43.091 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:28.787 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:43.107 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:28.816 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:43.081 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:28.819 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:43.084 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:28.831 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:43.096 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:28.846 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:43.166 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:28.848 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:43.168 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:28.862 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:43.181 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:38.781 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:53.100 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:38.798 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:53.117 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:38.813 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:26:53.132 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:38.832 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:53.097 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:38.834 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:53.099 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:38.835 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:53.155 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:38.837 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:53.157 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:38.846 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:53.110 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:38.850 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:26:53.169 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:53.817 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:08.136 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:53.821 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:08.140 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:53.840 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:08.159 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:53.872 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:08.136 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:53.873 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:08.193 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:53.874 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:08.139 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:53.876 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:08.196 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:53.891 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:08.155 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:53.895 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:08.215 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:58.833 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:13.152 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:58.835 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:13.154 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:58.848 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:13.167 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:58.884 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:13.148 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:58.885 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:13.150 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:58.895 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:13.214 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:58.896 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:13.161 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:58.899 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:13.218 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:45:58.912 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:13.232 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:03.861 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:18.180 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:03.866 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:18.186 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:03.882 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:18.201 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:03.913 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:18.177 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:03.915 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:18.180 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:03.923 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:18.242 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:03.928 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:18.192 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:03.928 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:18.248 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:03.942 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:18.262 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:13.884 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:28.203 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:13.887 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:28.206 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:13.902 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:28.221 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:13.931 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:28.196 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:13.933 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:28.198 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:13.943 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:28.262 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:13.946 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:28.211 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:13.948 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:28.267 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:13.961 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:28.280 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:18.908 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:33.227 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:18.913 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:33.232 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:18.927 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:33.246 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:18.957 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:33.222 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:18.960 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:33.225 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:18.963 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:33.283 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:18.969 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:33.288 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:18.971 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:33.235 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:18.982 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:33.301 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:23.934 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:38.253 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:23.938 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:38.258 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:23.953 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:38.272 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:23.983 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:38.247 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:23.986 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:38.250 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:23.988 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:38.307 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:23.992 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:38.311 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:23.998 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:38.263 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:24.008 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:38.327 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:33.958 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:48.277 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:33.963 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:48.282 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:33.979 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:48.298 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:34.010 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:48.274 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:34.012 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:48.331 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:34.012 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:48.276 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:34.019 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:48.338 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:34.025 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:48.289 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:34.033 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:48.352 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:38.971 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:53.290 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:38.990 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:53.309 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:39.004 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:27:53.323 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:39.025 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:53.289 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:39.028 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:53.292 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:39.039 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:53.358 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:39.040 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:53.304 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:39.045 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:53.364 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:39.056 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:27:53.375 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:54.015 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:08.331 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:54.016 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:08.335 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:54.034 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:08.353 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:54.066 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:08.330 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:54.067 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:08.331 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:54.071 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:08.390 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:54.073 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:08.392 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:54.083 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:08.347 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:54.090 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:08.409 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:59.025 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:13.344 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:59.029 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:13.348 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:59.042 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:13.361 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:59.082 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:13.346 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:59.083 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:13.402 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:59.083 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:13.347 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:59.086 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:13.404 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:59.094 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:13.358 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:46:59.097 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:13.416 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:14.090 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:28.408 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:14.095 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:28.413 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:14.138 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:28.457 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:14.147 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:28.411 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:14.149 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:28.413 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:14.154 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:28.473 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:14.158 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:28.476 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:14.165 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:28.429 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:14.175 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:28.493 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.117 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:33.435 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.121 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:33.440 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.137 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:33.455 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.169 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:33.433 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.173 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:33.437 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.186 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:33.450 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.186 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:33.504 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.192 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:33.510 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.206 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:33.525 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.794 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:34.058 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.795 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:34.059 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.803 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:34.066 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.846 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:34.165 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.847 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:34.166 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.856 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:34.174 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.877 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:34.195 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.878 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:34.197 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:19.887 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:34.205 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:34.179 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:48.497 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:34.183 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:48.502 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:34.201 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:48.519 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:34.234 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:48.498 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:34.239 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:48.501 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:34.239 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:48.558 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:34.241 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:48.560 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:34.253 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:48.516 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:34.263 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:48.581 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:39.170 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:53.488 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:39.174 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:53.492 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:39.187 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:28:53.505 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:39.222 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:53.485 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:39.225 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:53.488 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:39.225 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:53.543 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:39.228 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:53.547 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:39.237 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:53.500 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:39.247 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:28:53.565 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.231 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:08.549 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.237 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:08.555 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.256 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:08.574 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.290 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:08.553 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.291 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:08.555 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.292 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:08.610 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.296 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:08.614 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.308 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:08.571 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.314 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:08.632 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.884 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:09.147 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.885 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:09.149 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.893 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:09.156 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.943 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:09.261 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.944 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:09.262 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:54.957 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:09.275 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:55.013 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:09.331 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:55.014 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:09.332 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:55.023 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:09.341 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:59.244 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:13.562 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:59.252 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:13.570 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:59.267 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:13.585 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:59.300 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:13.618 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:59.304 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:13.622 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:59.314 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:13.578 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:59.316 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:13.579 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:59.318 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:13.636 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:47:59.326 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:13.589 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:14.317 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:28.635 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:14.319 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:28.637 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:14.335 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:28.653 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:14.365 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:28.628 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:14.368 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:28.686 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:14.369 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:28.632 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:14.373 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:28.691 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:14.386 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:28.649 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:14.389 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:28.707 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:15.021 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:29.284 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:15.022 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:29.285 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:15.024 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:29.342 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:15.025 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:29.343 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:15.034 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:29.298 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:15.038 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:29.356 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:15.082 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:29.400 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:15.083 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:29.401 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:15.096 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:29.414 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:19.300 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:33.618 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:19.305 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:33.623 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:19.317 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:33.635 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:19.351 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:33.615 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:19.353 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:33.671 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:19.353 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:33.616 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:19.358 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:33.676 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:19.363 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:33.627 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:19.369 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:33.687 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:34.373 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:48.690 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:34.378 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:48.695 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:34.397 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:48.715 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:34.430 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:48.693 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:34.432 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:48.695 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:34.433 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:48.751 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:34.436 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:48.754 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:34.449 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:48.712 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:34.453 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:48.771 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:39.354 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:53.672 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:39.357 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:53.675 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:39.369 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:29:53.687 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:39.403 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:53.666 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:39.406 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:53.723 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:39.406 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:53.669 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:39.409 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:53.727 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:39.417 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:53.679 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:39.421 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:29:53.738 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:54.392 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:08.710 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:54.394 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:08.712 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:54.412 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:08.730 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:54.438 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:08.700 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:54.441 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:08.704 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:54.449 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:08.766 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:54.451 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:08.768 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:54.457 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:08.720 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:54.467 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:08.785 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:59.407 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:13.724 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:59.413 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:13.731 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:59.428 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:13.746 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:59.464 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:13.781 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:59.468 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:13.786 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:59.479 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:13.797 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:59.483 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:13.746 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:59.485 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:13.747 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:48:59.494 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:13.757 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:14.444 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:28.761 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:14.446 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:28.764 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:14.463 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:28.781 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:14.492 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:28.755 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:14.493 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:28.810 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:14.496 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:28.758 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:14.500 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:28.817 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:14.514 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:28.776 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:14.518 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:28.835 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:19.456 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:33.773 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:19.461 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:33.779 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:19.474 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:33.792 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:19.513 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:33.775 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:19.514 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:33.777 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:19.515 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:33.832 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:19.518 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:33.835 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:19.525 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:33.788 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:19.531 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:33.848 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:39.520 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:53.837 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:39.523 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:53.840 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:39.547 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:53.864 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:39.575 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:53.838 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:39.576 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:53.893 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:39.577 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:53.839 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:39.579 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:53.896 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:39.598 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:53.860 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:39.601 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:53.918 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:40.036 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:54.298 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:40.038 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:54.300 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:40.041 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:54.358 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:40.046 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:54.308 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:40.057 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:54.374 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:40.066 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:30:54.383 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:40.115 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:54.432 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:40.116 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:54.433 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:40.127 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:30:54.444 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:59.567 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:13.884 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:59.571 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:13.889 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:59.592 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:13.909 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:59.624 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:13.940 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:59.624 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:13.885 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:59.625 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:13.887 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:59.626 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:13.943 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:59.642 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:13.905 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:49:59.647 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:13.964 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:00.231 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:14.493 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:00.232 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:14.494 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:00.240 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:14.502 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:00.305 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:14.623 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:00.307 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:14.624 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:00.318 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:14.635 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:00.325 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:14.642 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:00.326 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:14.643 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:00.337 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:14.654 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:19.618 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:33.934 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:19.622 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:33.938 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:19.641 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:33.958 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:19.673 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:33.934 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:19.674 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:33.936 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:19.676 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:33.993 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:19.679 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:33.996 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:19.693 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:33.955 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:19.705 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:34.022 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:20.283 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:34.545 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:20.283 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:34.546 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:20.291 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:34.553 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:20.347 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:34.664 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:20.348 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:34.665 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:20.358 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:34.673 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:20.358 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:34.674 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:20.358 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:34.675 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:20.370 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:34.686 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:39.626 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:53.942 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:39.678 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:53.940 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:39.681 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:53.943 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:39.682 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:53.998 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:39.703 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:53.965 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:39.705 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:54.022 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:39.737 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:54.054 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:39.741 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:54.057 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:39.760 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:54.077 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:40.291 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:54.553 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:40.292 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:54.554 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:40.300 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:54.562 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:40.379 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:54.696 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:40.380 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:54.697 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:40.391 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:31:54.708 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:40.414 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:54.731 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:40.416 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:54.732 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:40.424 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:31:54.740 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:59.731 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:14.047 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:59.734 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:14.051 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:59.756 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:14.072 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:59.783 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:14.045 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:59.786 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:14.048 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:59.788 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:14.105 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:59.792 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:14.109 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:59.805 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:14.067 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:50:59.813 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:14.130 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:00.400 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:14.662 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:00.401 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:14.663 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:00.409 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:14.671 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:00.433 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:14.749 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:00.434 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:14.751 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:00.443 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:14.759 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:00.468 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:14.784 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:00.469 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:14.786 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:00.478 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:14.794 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:19.828 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:34.090 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:19.830 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:34.146 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:19.834 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:34.095 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:19.834 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:34.150 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:19.894 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:34.155 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:19.900 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:34.216 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:19.901 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:34.218 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:19.907 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:34.223 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:19.955 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:34.271 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:20.548 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:34.810 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:20.549 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:34.811 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:20.557 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:34.819 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:20.646 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:34.962 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:20.647 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:34.964 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:20.655 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:34.972 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:20.696 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:35.013 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:20.698 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:35.015 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:20.766 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:35.083 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:21.314 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:35.576 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:21.320 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:35.582 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:21.394 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:35.655 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:21.569 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:35.885 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:21.570 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:35.886 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:21.585 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:35.902 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:21.669 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:35.985 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:21.670 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:35.986 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:21.684 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:36.000 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:35.107 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:49.423 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:35.128 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:49.444 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:35.128 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:49.445 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:35.133 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:49.449 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:35.134 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:49.395 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:35.137 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:49.398 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:35.239 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:49.500 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:35.241 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:49.558 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:35.267 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:49.583 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:36.085 INFO:tasks.workunit.client.0.smithi007.stdout: ec_io: Running main() from gmock_main.cc 2024-01-21T03:51:36.085 INFO:tasks.workunit.client.0.smithi007.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-01-21T03:51:36.085 INFO:tasks.workunit.client.0.smithi007.stdout: ec_io: [----------] Global test environment set-up. 2024-01-21T03:51:36.085 INFO:tasks.workunit.client.0.smithi007.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-01-21T03:51:36.085 INFO:tasks.workunit.client.0.smithi007.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-01-21T03:51:36.085 INFO:tasks.workunit.client.0.smithi007.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (10548 ms) 2024-01-21T03:51:36.085 INFO:tasks.workunit.client.0.smithi007.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-01-21T03:51:36.085 INFO:tasks.workunit.client.0.smithi007.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (1824680 ms) 2024-01-21T03:51:36.085 INFO:tasks.workunit.client.0.smithi007.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (1835228 ms total) 2024-01-21T03:51:36.085 INFO:tasks.workunit.client.0.smithi007.stdout: ec_io: 2024-01-21T03:51:36.086 INFO:tasks.workunit.client.0.smithi007.stdout: ec_io: [----------] Global test environment tear-down 2024-01-21T03:51:36.086 INFO:tasks.workunit.client.0.smithi007.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (1835228 ms total) 2024-01-21T03:51:36.086 INFO:tasks.workunit.client.0.smithi007.stdout: ec_io: [ PASSED ] 2 tests. 2024-01-21T03:51:36.087 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:51:36.087 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40253 2024-01-21T03:51:36.087 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40253 2024-01-21T03:51:36.087 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:51:36.087 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40883 2024-01-21T03:51:36.087 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40883 2024-01-21T03:51:36.087 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:51:36.087 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40751 2024-01-21T03:51:36.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40751 2024-01-21T03:51:36.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:51:36.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40122 2024-01-21T03:51:36.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40122 2024-01-21T03:51:36.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:51:36.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40186 2024-01-21T03:51:36.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40186 2024-01-21T03:51:36.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:51:36.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40782 2024-01-21T03:51:36.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40782 2024-01-21T03:51:36.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:51:36.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40113 2024-01-21T03:51:36.088 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40113 2024-01-21T03:51:39.925 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:54.241 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:39.986 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:54.302 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:40.017 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:32:54.333 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:40.043 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:54.305 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:40.047 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:54.363 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:40.050 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:54.366 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:40.053 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:54.314 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:40.069 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:32:54.385 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:40.082 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:54.343 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:40.670 INFO:tasks.workunit.client.0.smithi007.stdout: misc: Running main() from gmock_main.cc 2024-01-21T03:51:40.670 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [----------] Global test environment set-up. 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ OK ] NeoRadosMisc.Version (4203 ms) 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (3222 ms) 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ OK ] NeoRadosMisc.LongName (1802466 ms) 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3267 ms) 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3314 ms) 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-01-21T03:51:40.671 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3661 ms) 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ OK ] NeoRadosMisc.Exec (3496 ms) 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (2598 ms) 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3430 ms) 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ OK ] NeoRadosMisc.BigObject (4348 ms) 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2219 ms) 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3540 ms) 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [----------] 12 tests from NeoRadosMisc (1839764 ms total) 2024-01-21T03:51:40.672 INFO:tasks.workunit.client.0.smithi007.stdout: misc: 2024-01-21T03:51:40.673 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [----------] Global test environment tear-down 2024-01-21T03:51:40.673 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [==========] 12 tests from 1 test suite ran. (1839765 ms total) 2024-01-21T03:51:40.673 INFO:tasks.workunit.client.0.smithi007.stdout: misc: [ PASSED ] 12 tests. 2024-01-21T03:51:50.155 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:04.471 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:50.158 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:04.474 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:50.216 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:04.532 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:50.225 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:04.541 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:50.226 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:04.542 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:50.233 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:04.495 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:50.242 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:04.503 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:50.260 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:04.576 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:51:50.277 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:04.538 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:00.160 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:14.476 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:00.257 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:14.573 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:00.297 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:14.613 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:00.320 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:14.581 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:00.325 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:14.586 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:00.352 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:14.613 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:00.364 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:14.680 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:00.374 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:14.689 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:00.417 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:14.733 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:05.158 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:19.419 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:05.159 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:19.421 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:05.174 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:19.435 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:05.176 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:19.492 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:05.179 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:19.495 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:05.201 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:19.517 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:05.243 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:19.559 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:05.251 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:19.567 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:05.270 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:19.586 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:05.752 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:20.013 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:05.753 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:20.014 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:05.771 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:20.032 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:06.004 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:20.320 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:06.005 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:20.321 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:06.021 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:20.337 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:06.041 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:20.357 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:06.042 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:20.358 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:06.052 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:20.368 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:20.326 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:34.642 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:20.396 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:34.712 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:20.442 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:34.758 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:20.445 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:34.706 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:20.452 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:34.713 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:20.465 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:34.781 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:20.469 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:34.785 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:20.490 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:34.751 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:20.513 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:34.829 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:35.381 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:49.697 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:35.383 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:49.699 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:35.419 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:49.735 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:35.440 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:49.756 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:35.443 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:49.759 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:35.446 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:49.707 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:35.452 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:49.713 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:35.481 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:49.797 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:35.495 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:49.756 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:40.227 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:54.543 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:40.279 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:54.595 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:40.298 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:33:54.613 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:40.327 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:54.587 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:40.329 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:54.590 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:40.338 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:54.654 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:40.341 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:33:54.602 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:40.342 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:54.658 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:40.357 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:33:54.673 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:50.571 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:04.886 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:50.579 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:04.895 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:50.604 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:04.920 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:50.632 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:04.893 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:50.645 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:04.906 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:50.647 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:04.963 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:50.649 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:04.965 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:50.674 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:04.935 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:50.675 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:04.990 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:52:58.136 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-01-21T03:52:58.136 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-01-21T03:52:58.136 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-01-21T03:52:58.136 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (6180 ms) 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (1825989 ms) 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6868 ms) 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3574 ms) 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3385 ms) 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3330 ms) 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-01-21T03:52:58.137 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2524 ms) 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3144 ms) 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3263 ms) 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2708 ms) 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3305 ms) 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3440 ms) 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3398 ms) 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3399 ms) 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3436 ms) 2024-01-21T03:52:58.138 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3044 ms) 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3259 ms) 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3273 ms) 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3532 ms) 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3498 ms) 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2457 ms) 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3631 ms) 2024-01-21T03:52:58.139 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-01-21T03:52:58.140 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3278 ms) 2024-01-21T03:52:58.140 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-01-21T03:52:58.140 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3325 ms) 2024-01-21T03:52:58.140 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-01-21T03:52:58.140 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3808 ms) 2024-01-21T03:52:58.140 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-01-21T03:52:58.140 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3367 ms) 2024-01-21T03:52:58.140 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-01-21T03:53:00.599 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:14.915 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:00.607 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:14.923 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:00.644 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:14.959 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:00.670 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:14.986 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:00.672 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:14.987 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:00.675 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:14.936 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:00.681 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:14.942 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:00.698 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:15.014 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:00.711 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:14.972 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:05.441 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:19.756 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:05.458 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:19.773 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:05.478 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:19.794 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:05.487 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:19.748 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:05.491 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:19.751 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:05.500 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:19.816 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:05.501 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:19.817 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:05.504 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:19.765 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:05.521 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:19.837 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:14.210 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:28.526 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:14.215 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:28.531 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:14.230 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:28.491 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:14.234 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:28.494 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:14.263 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:28.523 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:14.266 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:28.582 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:14.289 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:28.605 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:14.291 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:28.607 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:14.321 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:28.637 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:20.481 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:34.796 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:20.485 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:34.800 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:20.504 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:34.819 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:20.529 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:34.789 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:20.530 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:34.791 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:20.537 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:34.853 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:20.539 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:34.855 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:20.546 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:34.807 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:20.560 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:34.876 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:21.261 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:35.521 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:21.262 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:35.522 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:21.272 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:35.532 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:21.362 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:35.678 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:21.363 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:35.679 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:21.373 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:35.689 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:21.375 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:35.691 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:21.389 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:35.705 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:21.397 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:35.713 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:35.539 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:49.855 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:35.544 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:49.860 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:35.574 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:34:49.890 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:35.592 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:49.852 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:35.595 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:49.856 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:35.603 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:49.918 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:35.604 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:49.920 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:35.622 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:34:49.883 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:35.633 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:34:49.948 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:50.617 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:04.932 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:50.625 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:04.941 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:50.668 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:04.984 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:50.674 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:04.934 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:50.677 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:04.938 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:50.682 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:04.998 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:50.684 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:04.999 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:50.709 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:04.969 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:53:50.719 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:05.035 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:05.665 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:19.980 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:05.667 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:19.982 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:05.704 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:20.019 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:05.715 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:19.976 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:05.716 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:19.977 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:05.725 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:20.041 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:05.736 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:20.052 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:05.746 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:20.006 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:05.782 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:20.098 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:20.701 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:35.016 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:20.705 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:35.020 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:20.733 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:35.048 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:20.751 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:35.011 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:20.752 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:35.012 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:20.755 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:35.071 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:20.759 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:35.075 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:20.779 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:35.039 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:20.797 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:35.112 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:21.540 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:35.800 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:21.541 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:35.802 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:21.555 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:35.815 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:21.648 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:35.963 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:21.649 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:35.965 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:21.663 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:35.978 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:21.673 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:35.988 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:21.674 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:35.989 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:21.686 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:36.002 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:41.054 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:55.368 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:41.058 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:55.373 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:41.108 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:35:55.423 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:41.114 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:55.375 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:41.120 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:55.380 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:41.120 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:55.435 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:41.123 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:55.439 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:41.163 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:35:55.424 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:41.166 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:35:55.481 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:51.090 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:05.405 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:51.093 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:05.408 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:51.115 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:05.431 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:51.139 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:05.399 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:51.142 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:05.403 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:51.145 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:05.460 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:51.148 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:05.463 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:51.169 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:05.429 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:54:51.175 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:05.490 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:01.154 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:15.469 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:01.159 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:15.474 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:01.182 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:15.497 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:01.210 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:15.525 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:01.211 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:15.527 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:01.217 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:15.477 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:01.219 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:15.479 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:01.242 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:15.503 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:01.245 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:15.560 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:06.018 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:20.332 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:06.023 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:20.338 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:06.038 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:20.353 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:06.068 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:20.329 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:06.071 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:20.331 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:06.073 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:20.388 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:06.081 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:20.396 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:06.085 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:20.345 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:06.095 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:20.410 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:06.854 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:21.114 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:06.855 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:21.115 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:06.868 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:21.128 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:07.079 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:21.393 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:07.080 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:21.395 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:07.099 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:21.413 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:07.102 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:21.417 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:07.103 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:21.418 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:07.117 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:21.432 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:21.067 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:35.381 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:21.071 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:35.386 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:21.124 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:35.439 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:21.126 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:35.386 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:21.128 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:35.389 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:21.136 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:35.451 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:21.140 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:35.455 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:21.158 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:35.418 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:21.168 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:35.483 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:26.036 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:40.351 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:26.082 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:40.342 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:26.083 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:40.343 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:26.086 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:40.401 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:26.087 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:40.402 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:26.088 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:40.403 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:26.099 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:40.359 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:26.104 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:40.419 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:26.115 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:40.430 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:36.121 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:50.436 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:36.126 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:50.441 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:36.151 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:36:50.466 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:36.175 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:50.435 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:36.177 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:50.492 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:36.178 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:50.438 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:36.179 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:50.494 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:36.200 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:36:50.460 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:36.218 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:36:50.532 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:40.199 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripAppendPP (3288 ms) 2024-01-21T03:55:40.200 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-01-21T03:55:40.200 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3354 ms) 2024-01-21T03:55:40.200 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-01-21T03:55:40.200 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3462 ms) 2024-01-21T03:55:40.200 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-01-21T03:55:40.200 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3347 ms) 2024-01-21T03:55:40.200 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-01-21T03:55:40.200 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: setting pool EIO 2024-01-21T03:55:40.200 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: max_success 0, min_failed 1 2024-01-21T03:55:40.200 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (5837 ms) 2024-01-21T03:55:40.200 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-01-21T03:55:40.200 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4496 ms) 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (1938200 ms total) 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3557 ms) 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9526 ms) 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (16237 ms) 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3447 ms) 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (32767 ms total) 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-01-21T03:55:40.201 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3299 ms) 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3299 ms total) 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (15187 ms) 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6836 ms) 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7801 ms) 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (8537 ms) 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3272 ms) 2024-01-21T03:55:40.202 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7774 ms) 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7035 ms) 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7279 ms) 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (8256 ms) 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (8057 ms) 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7092 ms) 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7663 ms) 2024-01-21T03:55:40.203 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-01-21T03:55:40.204 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7428 ms) 2024-01-21T03:55:40.204 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-01-21T03:55:46.278 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:00.592 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:46.280 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:00.595 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:46.309 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:00.624 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:46.331 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:00.591 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:46.336 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:00.596 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:46.356 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:00.671 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:46.359 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:00.619 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:46.363 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:00.678 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:46.391 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:00.706 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:51.174 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:05.489 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:51.177 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:05.492 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:51.194 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:05.508 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:51.225 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:05.485 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:51.226 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:05.486 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:51.238 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:05.498 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:51.252 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:05.566 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:51.253 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:05.568 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:55:51.266 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:05.581 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:06.286 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:20.600 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:06.290 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:20.605 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:06.321 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:20.636 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:06.342 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:20.602 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:06.344 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:20.603 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:06.354 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:20.668 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:06.357 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:20.671 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:06.371 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:20.631 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:06.402 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:20.716 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:07.232 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:21.492 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:07.234 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:21.494 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:07.253 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:21.513 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:07.412 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:21.726 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:07.412 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:21.727 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:07.421 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:21.736 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:07.423 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:21.737 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:07.423 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:21.737 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:07.434 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:21.749 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:21.343 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:35.658 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:21.348 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:35.663 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:21.380 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:35.694 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:21.395 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:35.655 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:21.396 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:35.656 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:21.402 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:35.717 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:21.405 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:35.720 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:21.424 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:35.683 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:21.433 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:35.747 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:25.158 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3282 ms) 2024-01-21T03:56:25.158 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-01-21T03:56:25.158 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7012 ms) 2024-01-21T03:56:25.158 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-01-21T03:56:25.158 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7804 ms) 2024-01-21T03:56:25.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-01-21T03:56:25.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7459 ms) 2024-01-21T03:56:25.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-01-21T03:56:25.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7738 ms) 2024-01-21T03:56:25.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-01-21T03:56:25.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7345 ms) 2024-01-21T03:56:25.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-01-21T03:56:25.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7597 ms) 2024-01-21T03:56:25.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (150455 ms total) 2024-01-21T03:56:25.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: 2024-01-21T03:56:25.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-01-21T03:56:25.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (2124722 ms total) 2024-01-21T03:56:25.159 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40134 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40134 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41152 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 41152 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40372 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40372 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41234 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 41234 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40468 2024-01-21T03:56:25.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40468 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40968 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40968 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41311 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 41311 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40401 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40401 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40807 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40807 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.164 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41124 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 41124 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40920 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40920 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41204 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 41204 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40734 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40734 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40297 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40297 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41290 2024-01-21T03:56:25.165 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 41290 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40610 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40610 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40106 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40106 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40165 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40165 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40552 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40552 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40206 2024-01-21T03:56:25.166 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40206 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40646 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40646 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40694 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40694 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41029 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 41029 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=41331 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 41331 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ exit 0 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ cleanup 2024-01-21T03:56:25.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ pkill -P 40101 2024-01-21T03:56:25.174 INFO:tasks.workunit.client.0.smithi007.stderr:+ true 2024-01-21T03:56:25.174 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-01-21T03:56:25.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-01-21T03:56:25.203 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-01-21T03:56:25.203 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-01-21T03:56:25.533 DEBUG:teuthology.parallel:result is None 2024-01-21T03:56:25.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-01-21T03:56:25.559 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-01-21T03:56:25.559 DEBUG:teuthology.orchestra.run.smithi007:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-01-21T03:56:25.623 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-01-21T03:56:25.623 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-21T03:56:25.633 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-21T03:56:25.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T03:56:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.763+0000 7f7a97456640 10 monclient: get_monmap_and_config 2024-01-21T03:56:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.763+0000 7f7a97456640 10 monclient: build_initial_monmap 2024-01-21T03:56:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.763+0000 7f7a97456640 10 monclient: monmap: 2024-01-21T03:56:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:25.764467+0000 2024-01-21T03:56:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:25.764467+0000 2024-01-21T03:56:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:25.766 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:25.766 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:25.766 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:25.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 1 Processor -- start 2024-01-21T03:56:25.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 1 -- start start 2024-01-21T03:56:25.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 10 monclient: init 2024-01-21T03:56:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-01-21T03:56:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a900a89d0 addr v1:172.21.15.7:6789/0 2024-01-21T03:56:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a90142bc0 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7a900a9c50 0x7f7a900a8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 10 monclient(hunting): picked mon.noname-c con 0x7f7a900a9c50 addr v2:172.21.15.177:3300/0 2024-01-21T03:56:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a90142020 con 0x7f7a900a89d0 2024-01-21T03:56:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a900aa940 con 0x7f7a90142bc0 2024-01-21T03:56:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a900aac20 con 0x7f7a900a9c50 2024-01-21T03:56:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.764+0000 7f7a97456640 10 monclient(hunting): authenticate will time out at 3532.080623s 2024-01-21T03:56:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.765+0000 7f7a951f2640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7a900a89d0 0x7f7a901424b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58052/0 (socket says 172.21.15.7:58052) 2024-01-21T03:56:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.765+0000 7f7a951f2640 1 -- 172.21.15.7:0/1481097749 learned_addr learned my addr 172.21.15.7:0/1481097749 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.765+0000 7f7a959f3640 1 --2- 172.21.15.7:0/1481097749 >> v2:172.21.15.177:3300/0 conn(0x7f7a900a9c50 0x7f7a900a8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.765+0000 7f7a959f3640 10 monclient(hunting): get_auth_request con 0x7f7a900a9c50 auth_method 0 2024-01-21T03:56:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.765+0000 7f7a959f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.765+0000 7f7a959f3640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.765+0000 7f7a959f3640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.766+0000 7f7a959f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.766+0000 7f7a959f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.766+0000 7f7a959f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.767+0000 7f7a8ffff640 1 -- 172.21.15.7:0/1481097749 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051079420 0 0) 0x7f7a90142020 con 0x7f7a900a89d0 2024-01-21T03:56:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.767+0000 7f7a8ffff640 10 monclient(hunting): _negotiate 2024-01-21T03:56:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.767+0000 7f7a8ffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.767+0000 7f7a8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.767+0000 7f7a8ffff640 10 monclient(hunting): my global_id is 15819 2024-01-21T03:56:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.767+0000 7f7a8ffff640 1 -- 172.21.15.7:0/1481097749 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a78003520 con 0x7f7a900a89d0 2024-01-21T03:56:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.768+0000 7f7a959f3640 10 monclient(hunting): handle_auth_done global_id 23134 payload 274 2024-01-21T03:56:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.768+0000 7f7a959f3640 1 -- 172.21.15.7:0/1481097749 >> v1:172.21.15.177:6789/0 conn(0x7f7a90142bc0 legacy=0x7f7a9013f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.768+0000 7f7a959f3640 1 -- 172.21.15.7:0/1481097749 >> v1:172.21.15.7:6789/0 conn(0x7f7a900a89d0 legacy=0x7f7a901424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.768+0000 7f7a959f3640 10 monclient: _finish_hunting 0 2024-01-21T03:56:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.768+0000 7f7a959f3640 1 monclient: found mon.noname-c 2024-01-21T03:56:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.768+0000 7f7a959f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.768+0000 7f7a959f3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:56:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.768+0000 7f7a959f3640 1 -- 172.21.15.7:0/1481097749 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a900b2230 con 0x7f7a900a9c50 2024-01-21T03:56:25.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.771+0000 7f7a959f3640 1 --2- 172.21.15.7:0/1481097749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a900a9c50 0x7f7a900a8db0 secure :-1 s=READY pgs=5803 cs=0 l=1 rev1=1 crypto rx=0x7f7a880097a0 tx=0x7f7a8802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=582c442c23afdb68 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a8ffff640 1 -- 172.21.15.7:0/1481097749 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8802f050 con 0x7f7a900a9c50 2024-01-21T03:56:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a8ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:56:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a8ffff640 10 monclient: dump: 2024-01-21T03:56:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a8ffff640 10 monclient: _finish_auth 0 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a8ffff640 10 monclient: _check_auth_tickets 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a8ffff640 1 -- 172.21.15.7:0/1481097749 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a88031700 con 0x7f7a900a9c50 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a8ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a8ffff640 1 -- 172.21.15.7:0/1481097749 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88031a90 con 0x7f7a900a9c50 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a8ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a8ffff640 10 monclient: dump: 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:25.778 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a97456640 5 monclient: authenticate success, global_id 23134 2024-01-21T03:56:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a97456640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.775+0000 7f7a97456640 4 set_mon_vals no callback set 2024-01-21T03:56:25.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.776+0000 7f7a97456640 10 monclient: shutdown 2024-01-21T03:56:25.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.776+0000 7f7a97456640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:25.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.776+0000 7f7a97456640 1 -- 172.21.15.7:0/1481097749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a900a9c50 msgr2=0x7f7a900a8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.776+0000 7f7a97456640 1 --2- 172.21.15.7:0/1481097749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a900a9c50 0x7f7a900a8db0 secure :-1 s=READY pgs=5803 cs=0 l=1 rev1=1 crypto rx=0x7f7a880097a0 tx=0x7f7a8802b7a0 comp rx=0 tx=0).stop 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.776+0000 7f7a97456640 1 -- 172.21.15.7:0/1481097749 shutdown_connections 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.776+0000 7f7a97456640 1 --2- 172.21.15.7:0/1481097749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a900a9c50 0x7f7a900a8db0 unknown :-1 s=CLOSED pgs=5803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.776+0000 7f7a97456640 1 -- 172.21.15.7:0/1481097749 >> 172.21.15.7:0/1481097749 conn(0x7f7a90066eb0 msgr2=0x7f7a900b7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.776+0000 7f7a97456640 1 -- 172.21.15.7:0/1481097749 shutdown_connections 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.776+0000 7f7a97456640 1 -- 172.21.15.7:0/1481097749 wait complete. 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.777+0000 7f7a97456640 10 monclient: build_initial_monmap 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.777+0000 7f7a97456640 10 monclient: monmap: 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.777+0000 7f7a97456640 1 librados: starting msgr at 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.777+0000 7f7a97456640 1 librados: starting objecter 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.777+0000 7f7a97456640 1 Processor -- start 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.777+0000 7f7a97456640 1 -- start start 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.777+0000 7f7a97456640 1 librados: setting wanted keys 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.777+0000 7f7a97456640 1 librados: calling monclient init 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.777+0000 7f7a97456640 10 monclient: init 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a900a89d0 0x7f7a90139d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:25.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 10 monclient(hunting): picked mon.noname-c con 0x7f7a900a89d0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a900a9c50 0x7f7a9013b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 10 monclient(hunting): picked mon.noname-b con 0x7f7a900a9c50 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a90142bc0 0x7f7a900aced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 10 monclient(hunting): picked mon.noname-a con 0x7f7a90142bc0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9013bf20 con 0x7f7a90142bc0 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a900ac4f0 con 0x7f7a900a9c50 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a900ac7d0 con 0x7f7a900a89d0 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a97456640 10 monclient(hunting): authenticate will time out at 3532.093827s 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a959f3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a90142bc0 0x7f7a900aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a949f1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a900a9c50 0x7f7a9013b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a951f2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a900a89d0 0x7f7a90139d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a949f1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a900a9c50 0x7f7a9013b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:38526/0 (socket says 172.21.15.7:38526) 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a949f1640 1 -- 172.21.15.7:0/2270871283 learned_addr learned my addr 172.21.15.7:0/2270871283 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a949f1640 10 monclient(hunting): get_auth_request con 0x7f7a900a9c50 auth_method 0 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a949f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a949f1640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.778+0000 7f7a949f1640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a951f2640 10 monclient(hunting): get_auth_request con 0x7f7a900a89d0 auth_method 0 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a951f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a951f2640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a951f2640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a959f3640 10 monclient(hunting): get_auth_request con 0x7f7a90142bc0 auth_method 0 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a959f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a959f3640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a959f3640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a949f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a949f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a949f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a951f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a951f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a951f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a959f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a959f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.779+0000 7f7a959f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.782+0000 7f7a949f1640 10 monclient(hunting): handle_auth_done global_id 23137 payload 824 2024-01-21T03:56:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.782+0000 7f7a949f1640 1 -- 172.21.15.7:0/2270871283 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a900a89d0 msgr2=0x7f7a90139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.782+0000 7f7a949f1640 1 --2- 172.21.15.7:0/2270871283 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a900a89d0 0x7f7a90139d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.782+0000 7f7a949f1640 1 -- 172.21.15.7:0/2270871283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a90142bc0 msgr2=0x7f7a900aced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.782+0000 7f7a949f1640 1 --2- 172.21.15.7:0/2270871283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a90142bc0 0x7f7a900aced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.782+0000 7f7a949f1640 10 monclient: _finish_hunting 0 2024-01-21T03:56:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.782+0000 7f7a949f1640 1 monclient: found mon.noname-b 2024-01-21T03:56:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.782+0000 7f7a949f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.782+0000 7f7a949f1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:56:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.782+0000 7f7a949f1640 1 -- 172.21.15.7:0/2270871283 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a88009400 con 0x7f7a900a9c50 2024-01-21T03:56:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.784+0000 7f7a949f1640 1 --2- 172.21.15.7:0/2270871283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a900a9c50 0x7f7a9013b650 secure :-1 s=READY pgs=5804 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400b3c0 tx=0x7f7a8400b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.784+0000 7f7a8dffb640 10 client.?.objecter ms_handle_connect 0x7f7a900a9c50 2024-01-21T03:56:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.784+0000 7f7a8dffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8dffb640 1 -- 172.21.15.7:0/2270871283 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a84011830 con 0x7f7a900a9c50 2024-01-21T03:56:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8dffb640 10 monclient: dump: 2024-01-21T03:56:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8dffb640 10 monclient: _finish_auth 0 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8dffb640 10 monclient: _check_auth_tickets 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8dffb640 1 -- 172.21.15.7:0/2270871283 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a84011990 con 0x7f7a900a9c50 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8dffb640 1 -- 172.21.15.7:0/2270871283 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a84011d80 con 0x7f7a900a9c50 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8dffb640 10 monclient: dump: 2024-01-21T03:56:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a97456640 5 monclient: authenticate success, global_id 23137 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a97456640 10 monclient: _renew_subs 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a8f7fe640 4 set_mon_vals no callback set 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a97456640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a97456640 1 -- 172.21.15.7:0/2270871283 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a900acab0 con 0x7f7a900a9c50 2024-01-21T03:56:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a97456640 10 client.23137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a97456640 10 monclient: _renew_subs 2024-01-21T03:56:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.787+0000 7f7a97456640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.788+0000 7f7a97456640 1 -- 172.21.15.7:0/2270871283 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9013bb90 con 0x7f7a900a9c50 2024-01-21T03:56:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.788+0000 7f7a97456640 1 librados: init done 2024-01-21T03:56:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.788+0000 7f7a97456640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.788+0000 7f7a97456640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.788+0000 7f7a97456640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:25.788+0000 7f7a97456640 1 -- 172.21.15.7:0/2270871283 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9004e970 con 0x7f7a900a9c50 2024-01-21T03:56:26.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.028+0000 7f7a8dffb640 1 -- 172.21.15.7:0/2270871283 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f7a84004020 con 0x7f7a900a9c50 2024-01-21T03:56:26.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.029+0000 7f7a8dffb640 1 --2- 172.21.15.7:0/2270871283 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7a7c06c8b0 0x7f7a7c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:26.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.029+0000 7f7a951f2640 1 --2- 172.21.15.7:0/2270871283 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7a7c06c8b0 0x7f7a7c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:26.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.029+0000 7f7a951f2640 10 monclient: get_auth_request con 0x7f7a7c06c8b0 auth_method 0 2024-01-21T03:56:26.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.029+0000 7f7a951f2640 1 --2- 172.21.15.7:0/2270871283 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7a7c06c8b0 0x7f7a7c06ed70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7a80005c00 tx=0x7f7a80009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:26.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.029+0000 7f7a8dffb640 10 client.23137.objecter ms_handle_connect 0x7f7a7c06c8b0 2024-01-21T03:56:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.030+0000 7f7a8dffb640 1 -- 172.21.15.7:0/2270871283 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7f7a840586b0 con 0x7f7a900a9c50 2024-01-21T03:56:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.030+0000 7f7a8dffb640 10 client.23137.objecter ms_dispatch 0x7f7a90000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.030+0000 7f7a8dffb640 3 client.23137.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.030+0000 7f7a8dffb640 3 client.23137.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:26.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.030+0000 7f7a8dffb640 20 client.23137.objecter dump_active .. 0 homeless 2024-01-21T03:56:26.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.040+0000 7f7a8dffb640 1 -- 172.21.15.7:0/2270871283 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f7a840549f0 con 0x7f7a900a9c50 2024-01-21T03:56:26.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.040+0000 7f7a8dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:26.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.040+0000 7f7a8dffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:26.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.217+0000 7f7a97456640 1 -- 172.21.15.7:0/2270871283 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7a9013fac0 con 0x7f7a7c06c8b0 2024-01-21T03:56:26.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.222+0000 7f7a8dffb640 1 -- 172.21.15.7:0/2270871283 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151891 (secure 0 0 0) 0x7f7a80001f60 con 0x7f7a7c06c8b0 2024-01-21T03:56:26.224 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:56:26.225 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-01-21T03:56:26.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 10 librados: watch_flush enter 2024-01-21T03:56:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 10 librados: watch_flush exit 2024-01-21T03:56:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 20 client.23137.objecter shutdown clearing up homeless session... 2024-01-21T03:56:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 10 client.23137.objecter successfully canceled tick 2024-01-21T03:56:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 1 -- 172.21.15.7:0/2270871283 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7a7c06c8b0 msgr2=0x7f7a7c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 1 --2- 172.21.15.7:0/2270871283 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7a7c06c8b0 0x7f7a7c06ed70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7a80005c00 tx=0x7f7a80009040 comp rx=0 tx=0).stop 2024-01-21T03:56:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 10 monclient: shutdown 2024-01-21T03:56:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 1 -- 172.21.15.7:0/2270871283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a900a9c50 msgr2=0x7f7a9013b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 1 --2- 172.21.15.7:0/2270871283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a900a9c50 0x7f7a9013b650 secure :-1 s=READY pgs=5804 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400b3c0 tx=0x7f7a8400b880 comp rx=0 tx=0).stop 2024-01-21T03:56:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 1 -- 172.21.15.7:0/2270871283 shutdown_connections 2024-01-21T03:56:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 1 --2- 172.21.15.7:0/2270871283 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7a7c06c8b0 0x7f7a7c06ed70 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 1 --2- 172.21.15.7:0/2270871283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a90142bc0 0x7f7a900aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 1 --2- 172.21.15.7:0/2270871283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a900a9c50 0x7f7a9013b650 unknown :-1 s=CLOSED pgs=5804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 1 --2- 172.21.15.7:0/2270871283 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a900a89d0 0x7f7a90139d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 1 -- 172.21.15.7:0/2270871283 >> 172.21.15.7:0/2270871283 conn(0x7f7a90066eb0 msgr2=0x7f7a9013ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:26.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 1 -- 172.21.15.7:0/2270871283 shutdown_connections 2024-01-21T03:56:26.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 1 -- 172.21.15.7:0/2270871283 wait complete. 2024-01-21T03:56:26.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.224+0000 7f7a97456640 1 librados: shutdown 2024-01-21T03:56:26.236 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":2397,"stamp":"2024-01-21T03:56:25.091444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":600632,"kb_used_data":40240,"kb_used_omap":12,"kb_used_meta":560243,"kb_avail":754374088,"statfs":{"total":773094113280,"available":772479066112,"internally_reserved":0,"allocated":41205760,"data_stored":19255096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":573689393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":148,"apply_latency_ms":148,"commit_latency_ns":148000000,"apply_latency_ns":148000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.529263"},"pg_stats":[{"pgid":"14.1e","version":"0'0","reported_seq":1254,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.191591+0000","last_change":"2024-01-21T03:21:15.540562+0000","last_active":"2024-01-21T03:56:23.191591+0000","last_peered":"2024-01-21T03:56:23.191591+0000","last_clean":"2024-01-21T03:56:23.191591+0000","last_became_active":"2024-01-21T03:21:15.289479+0000","last_became_peered":"2024-01-21T03:21:15.289479+0000","last_unstale":"2024-01-21T03:56:23.191591+0000","last_undegraded":"2024-01-21T03:56:23.191591+0000","last_fullsized":"2024-01-21T03:56:23.191591+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:26:47.504093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1210,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.683250+0000","last_change":"2024-01-21T03:21:15.879415+0000","last_active":"2024-01-21T03:56:20.683250+0000","last_peered":"2024-01-21T03:56:20.683250+0000","last_clean":"2024-01-21T03:56:20.683250+0000","last_became_active":"2024-01-21T03:21:15.864718+0000","last_became_peered":"2024-01-21T03:21:15.864718+0000","last_unstale":"2024-01-21T03:56:20.683250+0000","last_undegraded":"2024-01-21T03:56:20.683250+0000","last_fullsized":"2024-01-21T03:56:20.683250+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:26:29.398180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1231,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.706988+0000","last_change":"2024-01-21T03:21:09.350719+0000","last_active":"2024-01-21T03:56:19.706988+0000","last_peered":"2024-01-21T03:56:19.706988+0000","last_clean":"2024-01-21T03:56:19.706988+0000","last_became_active":"2024-01-21T03:21:09.333873+0000","last_became_peered":"2024-01-21T03:21:09.333873+0000","last_unstale":"2024-01-21T03:56:19.706988+0000","last_undegraded":"2024-01-21T03:56:19.706988+0000","last_fullsized":"2024-01-21T03:56:19.706988+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:44:56.009576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1230,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.697115+0000","last_change":"2024-01-21T03:21:13.545831+0000","last_active":"2024-01-21T03:56:20.697115+0000","last_peered":"2024-01-21T03:56:20.697115+0000","last_clean":"2024-01-21T03:56:20.697115+0000","last_became_active":"2024-01-21T03:21:13.539461+0000","last_became_peered":"2024-01-21T03:21:13.539461+0000","last_unstale":"2024-01-21T03:56:20.697115+0000","last_undegraded":"2024-01-21T03:56:20.697115+0000","last_fullsized":"2024-01-21T03:56:20.697115+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:55:11.327991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1207,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.484590+0000","last_change":"2024-01-21T03:21:31.051184+0000","last_active":"2024-01-21T03:56:19.484590+0000","last_peered":"2024-01-21T03:56:19.484590+0000","last_clean":"2024-01-21T03:56:19.484590+0000","last_became_active":"2024-01-21T03:21:30.818768+0000","last_became_peered":"2024-01-21T03:21:30.818768+0000","last_unstale":"2024-01-21T03:56:19.484590+0000","last_undegraded":"2024-01-21T03:56:19.484590+0000","last_fullsized":"2024-01-21T03:56:19.484590+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:16:32.050608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1232,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.685544+0000","last_change":"2024-01-21T03:21:13.851032+0000","last_active":"2024-01-21T03:56:20.685544+0000","last_peered":"2024-01-21T03:56:20.685544+0000","last_clean":"2024-01-21T03:56:20.685544+0000","last_became_active":"2024-01-21T03:21:13.842398+0000","last_became_peered":"2024-01-21T03:21:13.842398+0000","last_unstale":"2024-01-21T03:56:20.685544+0000","last_undegraded":"2024-01-21T03:56:20.685544+0000","last_fullsized":"2024-01-21T03:56:20.685544+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:27:03.441063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1233,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.669592+0000","last_change":"2024-01-21T03:21:34.296238+0000","last_active":"2024-01-21T03:56:20.669592+0000","last_peered":"2024-01-21T03:56:20.669592+0000","last_clean":"2024-01-21T03:56:20.669592+0000","last_became_active":"2024-01-21T03:21:34.290896+0000","last_became_peered":"2024-01-21T03:21:34.290896+0000","last_unstale":"2024-01-21T03:56:20.669592+0000","last_undegraded":"2024-01-21T03:56:20.669592+0000","last_fullsized":"2024-01-21T03:56:20.669592+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:52:43.200440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1","version":"60'7","reported_seq":1224,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.681000+0000","last_change":"2024-01-21T03:21:35.777407+0000","last_active":"2024-01-21T03:56:20.681000+0000","last_peered":"2024-01-21T03:56:20.681000+0000","last_clean":"2024-01-21T03:56:20.681000+0000","last_became_active":"2024-01-21T03:21:35.763549+0000","last_became_peered":"2024-01-21T03:21:35.763549+0000","last_unstale":"2024-01-21T03:56:20.681000+0000","last_undegraded":"2024-01-21T03:56:20.681000+0000","last_fullsized":"2024-01-21T03:56:20.681000+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:56:17.661965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1019,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.545211+0000","last_change":"2024-01-21T03:26:06.764781+0000","last_active":"2024-01-21T03:56:19.545211+0000","last_peered":"2024-01-21T03:56:19.545211+0000","last_clean":"2024-01-21T03:56:19.545211+0000","last_became_active":"2024-01-21T03:26:06.758343+0000","last_became_peered":"2024-01-21T03:26:06.758343+0000","last_unstale":"2024-01-21T03:56:19.545211+0000","last_undegraded":"2024-01-21T03:56:19.545211+0000","last_fullsized":"2024-01-21T03:56:19.545211+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:09:42.970888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1226,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.259577+0000","last_change":"2024-01-21T03:21:15.810535+0000","last_active":"2024-01-21T03:56:20.259577+0000","last_peered":"2024-01-21T03:56:20.259577+0000","last_clean":"2024-01-21T03:56:20.259577+0000","last_became_active":"2024-01-21T03:21:15.779865+0000","last_became_peered":"2024-01-21T03:21:15.779865+0000","last_unstale":"2024-01-21T03:56:20.259577+0000","last_undegraded":"2024-01-21T03:56:20.259577+0000","last_fullsized":"2024-01-21T03:56:20.259577+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:56:54.491579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1207,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.502562+0000","last_change":"2024-01-21T03:21:22.661192+0000","last_active":"2024-01-21T03:56:19.502562+0000","last_peered":"2024-01-21T03:56:19.502562+0000","last_clean":"2024-01-21T03:56:19.502562+0000","last_became_active":"2024-01-21T03:21:22.647696+0000","last_became_peered":"2024-01-21T03:21:22.647696+0000","last_unstale":"2024-01-21T03:56:19.502562+0000","last_undegraded":"2024-01-21T03:56:19.502562+0000","last_fullsized":"2024-01-21T03:56:19.502562+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:05:43.481850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1218,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.275498+0000","last_change":"2024-01-21T03:21:26.231500+0000","last_active":"2024-01-21T03:56:20.275498+0000","last_peered":"2024-01-21T03:56:20.275498+0000","last_clean":"2024-01-21T03:56:20.275498+0000","last_became_active":"2024-01-21T03:21:26.223079+0000","last_became_peered":"2024-01-21T03:21:26.223079+0000","last_unstale":"2024-01-21T03:56:20.275498+0000","last_undegraded":"2024-01-21T03:56:20.275498+0000","last_fullsized":"2024-01-21T03:56:20.275498+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:36:33.965862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1232,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.261261+0000","last_change":"2024-01-21T03:26:06.699602+0000","last_active":"2024-01-21T03:56:20.261261+0000","last_peered":"2024-01-21T03:56:20.261261+0000","last_clean":"2024-01-21T03:56:20.261261+0000","last_became_active":"2024-01-21T03:26:06.693281+0000","last_became_peered":"2024-01-21T03:26:06.693281+0000","last_unstale":"2024-01-21T03:56:20.261261+0000","last_undegraded":"2024-01-21T03:56:20.261261+0000","last_fullsized":"2024-01-21T03:56:20.261261+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:29:07.025641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"23'76","reported_seq":1257,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.243138+0000","last_change":"2024-01-21T03:19:59.052204+0000","last_active":"2024-01-21T03:56:20.243138+0000","last_peered":"2024-01-21T03:56:20.243138+0000","last_clean":"2024-01-21T03:56:20.243138+0000","last_became_active":"2024-01-21T03:19:58.060095+0000","last_became_peered":"2024-01-21T03:19:58.060095+0000","last_unstale":"2024-01-21T03:56:20.243138+0000","last_undegraded":"2024-01-21T03:56:20.243138+0000","last_fullsized":"2024-01-21T03:56:20.243138+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:44.383554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:44.383554+0000","last_clean_scrub_stamp":"2024-01-21T03:19:44.383554+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:42:45.445502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1260,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.144254+0000","last_change":"2024-01-21T03:20:02.295222+0000","last_active":"2024-01-21T03:56:23.144254+0000","last_peered":"2024-01-21T03:56:23.144254+0000","last_clean":"2024-01-21T03:56:23.144254+0000","last_became_active":"2024-01-21T03:20:02.289915+0000","last_became_peered":"2024-01-21T03:20:02.289915+0000","last_unstale":"2024-01-21T03:56:23.144254+0000","last_undegraded":"2024-01-21T03:56:23.144254+0000","last_fullsized":"2024-01-21T03:56:23.144254+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:36:11.210361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1218,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.272174+0000","last_change":"2024-01-21T03:21:14.853643+0000","last_active":"2024-01-21T03:56:20.272174+0000","last_peered":"2024-01-21T03:56:20.272174+0000","last_clean":"2024-01-21T03:56:20.272174+0000","last_became_active":"2024-01-21T03:21:14.829810+0000","last_became_peered":"2024-01-21T03:21:14.829810+0000","last_unstale":"2024-01-21T03:56:20.272174+0000","last_undegraded":"2024-01-21T03:56:20.272174+0000","last_fullsized":"2024-01-21T03:56:20.272174+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:30:12.597754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1261,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.105259+0000","last_change":"2024-01-21T03:20:01.812917+0000","last_active":"2024-01-21T03:56:23.105259+0000","last_peered":"2024-01-21T03:56:23.105259+0000","last_clean":"2024-01-21T03:56:23.105259+0000","last_became_active":"2024-01-21T03:20:01.784281+0000","last_became_peered":"2024-01-21T03:20:01.784281+0000","last_unstale":"2024-01-21T03:56:23.105259+0000","last_undegraded":"2024-01-21T03:56:23.105259+0000","last_fullsized":"2024-01-21T03:56:23.105259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:26:56.084862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1254,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.175368+0000","last_change":"2024-01-21T03:21:26.074817+0000","last_active":"2024-01-21T03:56:23.175368+0000","last_peered":"2024-01-21T03:56:23.175368+0000","last_clean":"2024-01-21T03:56:23.175368+0000","last_became_active":"2024-01-21T03:21:26.057996+0000","last_became_peered":"2024-01-21T03:21:26.057996+0000","last_unstale":"2024-01-21T03:56:23.175368+0000","last_undegraded":"2024-01-21T03:56:23.175368+0000","last_fullsized":"2024-01-21T03:56:23.175368+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:22:40.041984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1216,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.674340+0000","last_change":"2024-01-21T03:20:01.846748+0000","last_active":"2024-01-21T03:56:20.674340+0000","last_peered":"2024-01-21T03:56:20.674340+0000","last_clean":"2024-01-21T03:56:20.674340+0000","last_became_active":"2024-01-21T03:20:01.818592+0000","last_became_peered":"2024-01-21T03:20:01.818592+0000","last_unstale":"2024-01-21T03:56:20.674340+0000","last_undegraded":"2024-01-21T03:56:20.674340+0000","last_fullsized":"2024-01-21T03:56:20.674340+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:08:09.861694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1226,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.257372+0000","last_change":"2024-01-21T03:22:06.056192+0000","last_active":"2024-01-21T03:56:20.257372+0000","last_peered":"2024-01-21T03:56:20.257372+0000","last_clean":"2024-01-21T03:56:20.257372+0000","last_became_active":"2024-01-21T03:22:06.040269+0000","last_became_peered":"2024-01-21T03:22:06.040269+0000","last_unstale":"2024-01-21T03:56:20.257372+0000","last_undegraded":"2024-01-21T03:56:20.257372+0000","last_fullsized":"2024-01-21T03:56:20.257372+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:36:24.196035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1260,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.122291+0000","last_change":"2024-01-21T03:20:02.210632+0000","last_active":"2024-01-21T03:56:23.122291+0000","last_peered":"2024-01-21T03:56:23.122291+0000","last_clean":"2024-01-21T03:56:23.122291+0000","last_became_active":"2024-01-21T03:20:02.201314+0000","last_became_peered":"2024-01-21T03:20:02.201314+0000","last_unstale":"2024-01-21T03:56:23.122291+0000","last_undegraded":"2024-01-21T03:56:23.122291+0000","last_fullsized":"2024-01-21T03:56:23.122291+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:55:53.424403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1232,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.668006+0000","last_change":"2024-01-21T03:21:34.314505+0000","last_active":"2024-01-21T03:56:20.668006+0000","last_peered":"2024-01-21T03:56:20.668006+0000","last_clean":"2024-01-21T03:56:20.668006+0000","last_became_active":"2024-01-21T03:21:34.307508+0000","last_became_peered":"2024-01-21T03:21:34.307508+0000","last_unstale":"2024-01-21T03:56:20.668006+0000","last_undegraded":"2024-01-21T03:56:20.668006+0000","last_fullsized":"2024-01-21T03:56:20.668006+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:38:13.727492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1224,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.273664+0000","last_change":"2024-01-21T03:20:02.214415+0000","last_active":"2024-01-21T03:56:20.273664+0000","last_peered":"2024-01-21T03:56:20.273664+0000","last_clean":"2024-01-21T03:56:20.273664+0000","last_became_active":"2024-01-21T03:20:02.207814+0000","last_became_peered":"2024-01-21T03:20:02.207814+0000","last_unstale":"2024-01-21T03:56:20.273664+0000","last_undegraded":"2024-01-21T03:56:20.273664+0000","last_fullsized":"2024-01-21T03:56:20.273664+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:12:28.359349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1211,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.542317+0000","last_change":"2024-01-21T03:21:23.773815+0000","last_active":"2024-01-21T03:56:19.542317+0000","last_peered":"2024-01-21T03:56:19.542317+0000","last_clean":"2024-01-21T03:56:19.542317+0000","last_became_active":"2024-01-21T03:21:23.767027+0000","last_became_peered":"2024-01-21T03:21:23.767027+0000","last_unstale":"2024-01-21T03:56:19.542317+0000","last_undegraded":"2024-01-21T03:56:19.542317+0000","last_fullsized":"2024-01-21T03:56:19.542317+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:56:41.913140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1260,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.171699+0000","last_change":"2024-01-21T03:20:01.878033+0000","last_active":"2024-01-21T03:56:23.171699+0000","last_peered":"2024-01-21T03:56:23.171699+0000","last_clean":"2024-01-21T03:56:23.171699+0000","last_became_active":"2024-01-21T03:20:01.872724+0000","last_became_peered":"2024-01-21T03:20:01.872724+0000","last_unstale":"2024-01-21T03:56:23.171699+0000","last_undegraded":"2024-01-21T03:56:23.171699+0000","last_fullsized":"2024-01-21T03:56:23.171699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:06:41.515884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1207,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.486663+0000","last_change":"2024-01-21T03:21:55.470689+0000","last_active":"2024-01-21T03:56:19.486663+0000","last_peered":"2024-01-21T03:56:19.486663+0000","last_clean":"2024-01-21T03:56:19.486663+0000","last_became_active":"2024-01-21T03:21:55.460120+0000","last_became_peered":"2024-01-21T03:21:55.460120+0000","last_unstale":"2024-01-21T03:56:19.486663+0000","last_undegraded":"2024-01-21T03:56:19.486663+0000","last_fullsized":"2024-01-21T03:56:19.486663+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:03:53.385543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1236,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.701374+0000","last_change":"2024-01-21T03:20:02.260437+0000","last_active":"2024-01-21T03:56:20.701374+0000","last_peered":"2024-01-21T03:56:20.701374+0000","last_clean":"2024-01-21T03:56:20.701374+0000","last_became_active":"2024-01-21T03:20:02.234092+0000","last_became_peered":"2024-01-21T03:20:02.234092+0000","last_unstale":"2024-01-21T03:56:20.701374+0000","last_undegraded":"2024-01-21T03:56:20.701374+0000","last_fullsized":"2024-01-21T03:56:20.701374+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:32:42.605439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1232,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.703783+0000","last_change":"2024-01-21T03:21:09.536570+0000","last_active":"2024-01-21T03:56:19.703783+0000","last_peered":"2024-01-21T03:56:19.703783+0000","last_clean":"2024-01-21T03:56:19.703783+0000","last_became_active":"2024-01-21T03:21:09.531293+0000","last_became_peered":"2024-01-21T03:21:09.531293+0000","last_unstale":"2024-01-21T03:56:19.703783+0000","last_undegraded":"2024-01-21T03:56:19.703783+0000","last_fullsized":"2024-01-21T03:56:19.703783+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:01:17.017524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1256,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.197738+0000","last_change":"2024-01-21T03:20:02.233166+0000","last_active":"2024-01-21T03:56:23.197738+0000","last_peered":"2024-01-21T03:56:23.197738+0000","last_clean":"2024-01-21T03:56:23.197738+0000","last_became_active":"2024-01-21T03:20:02.226975+0000","last_became_peered":"2024-01-21T03:20:02.226975+0000","last_unstale":"2024-01-21T03:56:23.197738+0000","last_undegraded":"2024-01-21T03:56:23.197738+0000","last_fullsized":"2024-01-21T03:56:23.197738+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:36:45.013749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1253,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.119814+0000","last_change":"2024-01-21T03:21:36.938379+0000","last_active":"2024-01-21T03:56:23.119814+0000","last_peered":"2024-01-21T03:56:23.119814+0000","last_clean":"2024-01-21T03:56:23.119814+0000","last_became_active":"2024-01-21T03:21:36.929227+0000","last_became_peered":"2024-01-21T03:21:36.929227+0000","last_unstale":"2024-01-21T03:56:23.119814+0000","last_undegraded":"2024-01-21T03:56:23.119814+0000","last_fullsized":"2024-01-21T03:56:23.119814+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:55:39.626168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1218,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.252018+0000","last_change":"2024-01-21T03:21:13.975490+0000","last_active":"2024-01-21T03:56:20.252018+0000","last_peered":"2024-01-21T03:56:20.252018+0000","last_clean":"2024-01-21T03:56:20.252018+0000","last_became_active":"2024-01-21T03:21:13.969066+0000","last_became_peered":"2024-01-21T03:21:13.969066+0000","last_unstale":"2024-01-21T03:56:20.252018+0000","last_undegraded":"2024-01-21T03:56:20.252018+0000","last_fullsized":"2024-01-21T03:56:20.252018+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:53:18.899854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1231,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.705306+0000","last_change":"2024-01-21T03:21:22.338167+0000","last_active":"2024-01-21T03:56:19.705306+0000","last_peered":"2024-01-21T03:56:19.705306+0000","last_clean":"2024-01-21T03:56:19.705306+0000","last_became_active":"2024-01-21T03:21:22.319120+0000","last_became_peered":"2024-01-21T03:21:22.319120+0000","last_unstale":"2024-01-21T03:56:19.705306+0000","last_undegraded":"2024-01-21T03:56:19.705306+0000","last_fullsized":"2024-01-21T03:56:19.705306+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:35:43.974713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1230,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.681744+0000","last_change":"2024-01-21T03:21:17.587236+0000","last_active":"2024-01-21T03:56:20.681744+0000","last_peered":"2024-01-21T03:56:20.681744+0000","last_clean":"2024-01-21T03:56:20.681744+0000","last_became_active":"2024-01-21T03:21:17.582134+0000","last_became_peered":"2024-01-21T03:21:17.582134+0000","last_unstale":"2024-01-21T03:56:20.681744+0000","last_undegraded":"2024-01-21T03:56:20.681744+0000","last_fullsized":"2024-01-21T03:56:20.681744+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:14:12.050155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1253,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.167852+0000","last_change":"2024-01-21T03:21:40.581757+0000","last_active":"2024-01-21T03:56:23.167852+0000","last_peered":"2024-01-21T03:56:23.167852+0000","last_clean":"2024-01-21T03:56:23.167852+0000","last_became_active":"2024-01-21T03:21:40.568805+0000","last_became_peered":"2024-01-21T03:21:40.568805+0000","last_unstale":"2024-01-21T03:56:23.167852+0000","last_undegraded":"2024-01-21T03:56:23.167852+0000","last_fullsized":"2024-01-21T03:56:23.167852+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:33:02.221661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1226,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.588370+0000","last_change":"2024-01-21T03:22:06.157185+0000","last_active":"2024-01-21T03:56:20.588370+0000","last_peered":"2024-01-21T03:56:20.588370+0000","last_clean":"2024-01-21T03:56:20.588370+0000","last_became_active":"2024-01-21T03:22:06.149457+0000","last_became_peered":"2024-01-21T03:22:06.149457+0000","last_unstale":"2024-01-21T03:56:20.588370+0000","last_undegraded":"2024-01-21T03:56:20.588370+0000","last_fullsized":"2024-01-21T03:56:20.588370+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:47:38.535776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1230,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.721033+0000","last_change":"2024-01-21T03:21:13.379783+0000","last_active":"2024-01-21T03:56:20.721033+0000","last_peered":"2024-01-21T03:56:20.721033+0000","last_clean":"2024-01-21T03:56:20.721033+0000","last_became_active":"2024-01-21T03:21:13.373828+0000","last_became_peered":"2024-01-21T03:21:13.373828+0000","last_unstale":"2024-01-21T03:56:20.721033+0000","last_undegraded":"2024-01-21T03:56:20.721033+0000","last_fullsized":"2024-01-21T03:56:20.721033+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:54:16.898824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1229,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.720120+0000","last_change":"2024-01-21T03:21:11.007665+0000","last_active":"2024-01-21T03:56:19.720120+0000","last_peered":"2024-01-21T03:56:19.720120+0000","last_clean":"2024-01-21T03:56:19.720120+0000","last_became_active":"2024-01-21T03:21:10.999264+0000","last_became_peered":"2024-01-21T03:21:10.999264+0000","last_unstale":"2024-01-21T03:56:19.720120+0000","last_undegraded":"2024-01-21T03:56:19.720120+0000","last_fullsized":"2024-01-21T03:56:19.720120+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:43:27.507683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1226,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.245990+0000","last_change":"2024-01-21T03:21:15.387706+0000","last_active":"2024-01-21T03:56:20.245990+0000","last_peered":"2024-01-21T03:56:20.245990+0000","last_clean":"2024-01-21T03:56:20.245990+0000","last_became_active":"2024-01-21T03:21:15.068960+0000","last_became_peered":"2024-01-21T03:21:15.068960+0000","last_unstale":"2024-01-21T03:56:20.245990+0000","last_undegraded":"2024-01-21T03:56:20.245990+0000","last_fullsized":"2024-01-21T03:56:20.245990+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:01:23.405734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1232,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.696894+0000","last_change":"2024-01-21T03:21:33.019724+0000","last_active":"2024-01-21T03:56:19.696894+0000","last_peered":"2024-01-21T03:56:19.696894+0000","last_clean":"2024-01-21T03:56:19.696894+0000","last_became_active":"2024-01-21T03:21:32.997061+0000","last_became_peered":"2024-01-21T03:21:32.997061+0000","last_unstale":"2024-01-21T03:56:19.696894+0000","last_undegraded":"2024-01-21T03:56:19.696894+0000","last_fullsized":"2024-01-21T03:56:19.696894+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:52:53.090792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1218,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.266819+0000","last_change":"2024-01-21T03:21:14.774673+0000","last_active":"2024-01-21T03:56:20.266819+0000","last_peered":"2024-01-21T03:56:20.266819+0000","last_clean":"2024-01-21T03:56:20.266819+0000","last_became_active":"2024-01-21T03:21:14.761665+0000","last_became_peered":"2024-01-21T03:21:14.761665+0000","last_unstale":"2024-01-21T03:56:20.266819+0000","last_undegraded":"2024-01-21T03:56:20.266819+0000","last_fullsized":"2024-01-21T03:56:20.266819+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:09:47.334671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1232,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.715724+0000","last_change":"2024-01-21T03:21:34.181963+0000","last_active":"2024-01-21T03:56:19.715724+0000","last_peered":"2024-01-21T03:56:19.715724+0000","last_clean":"2024-01-21T03:56:19.715724+0000","last_became_active":"2024-01-21T03:21:34.168272+0000","last_became_peered":"2024-01-21T03:21:34.168272+0000","last_unstale":"2024-01-21T03:56:19.715724+0000","last_undegraded":"2024-01-21T03:56:19.715724+0000","last_fullsized":"2024-01-21T03:56:19.715724+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:16:18.395527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280948,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76408,"kb_used_data":4964,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295432,"statfs":{"total":96636764160,"available":96558522368,"internally_reserved":0,"allocated":5083136,"data_stored":2325045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":9.3070000000000004,"5min":22.673999999999999,"15min":23.361000000000001},"min":{"1min":5.9930000000000003,"5min":4.7569999999999997,"15min":4.2290000000000001},"max":{"1min":15.645,"5min":221.726,"15min":236.40199999999999},"last":8.2629999999999999},{"interface":"front","average":{"1min":12.923,"5min":28.297999999999998,"15min":22.507000000000001},"min":{"1min":7.3280000000000003,"5min":6.5869999999999997,"15min":4.423},"max":{"1min":21.279,"5min":315.84899999999999,"15min":317.16500000000002},"last":11.234}]},{"osd":1,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":12.68,"5min":24.010999999999999,"15min":26.009},"min":{"1min":7.8280000000000003,"5min":7.8280000000000003,"15min":6.7649999999999997},"max":{"1min":19.681999999999999,"5min":210.08799999999999,"15min":688.34500000000003},"last":11.881},{"interface":"front","average":{"1min":15.066000000000001,"5min":30.968,"15min":31.401},"min":{"1min":10.843999999999999,"5min":10.49,"15min":9.6310000000000002},"max":{"1min":23.169,"5min":288.46600000000001,"15min":686.94299999999998},"last":17.687999999999999}]},{"osd":2,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":9.9570000000000007,"5min":59.222999999999999,"15min":34.966999999999999},"min":{"1min":6.0730000000000004,"5min":6.0730000000000004,"15min":5.75},"max":{"1min":31.401,"5min":3500.2379999999998,"15min":3500.2379999999998},"last":8.6159999999999997},{"interface":"front","average":{"1min":14.031000000000001,"5min":37.201000000000001,"15min":29.867000000000001},"min":{"1min":7.9669999999999996,"5min":7.9669999999999996,"15min":7.4710000000000001},"max":{"1min":25.942,"5min":919.58199999999999,"15min":919.58199999999999},"last":13.473000000000001}]},{"osd":4,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":14.010999999999999,"5min":26.649000000000001,"15min":26.817},"min":{"1min":10.026,"5min":9.1489999999999991,"15min":8.9909999999999997},"max":{"1min":18.762,"5min":241.95599999999999,"15min":297.46600000000001},"last":12.843999999999999},{"interface":"front","average":{"1min":18.001999999999999,"5min":31.356999999999999,"15min":31.646999999999998},"min":{"1min":12.919,"5min":12.346,"15min":9.673},"max":{"1min":23.687000000000001,"5min":292.25400000000002,"15min":340.73500000000001},"last":19.527999999999999}]},{"osd":5,"last update":"Sun Jan 21 03:56:18 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":24.042999999999999,"15min":24.727},"min":{"1min":7.3680000000000003,"5min":7.3680000000000003,"15min":6.1989999999999998},"max":{"1min":22.300000000000001,"5min":214.58500000000001,"15min":362.22899999999998},"last":14.112},{"interface":"front","average":{"1min":13.831,"5min":26.408000000000001,"15min":27.535},"min":{"1min":8.9079999999999995,"5min":6.5439999999999996,"15min":6.5439999999999996},"max":{"1min":21.576000000000001,"5min":339.19499999999999,"15min":339.19499999999999},"last":16.254999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":18.814,"5min":31.257000000000001,"15min":30.303999999999998},"min":{"1min":11.645,"5min":11.206,"15min":7.1719999999999997},"max":{"1min":42.293999999999997,"5min":211.822,"15min":572.20899999999995},"last":16.989000000000001},{"interface":"front","average":{"1min":21.713000000000001,"5min":36.789999999999999,"15min":34.954999999999998},"min":{"1min":13.535,"5min":13.535,"15min":9.7240000000000002},"max":{"1min":45.457999999999998,"5min":215.315,"15min":376.66399999999999},"last":20.728000000000002}]},{"osd":7,"last update":"Sun Jan 21 03:56:18 2024","interfaces":[{"interface":"back","average":{"1min":15.071999999999999,"5min":23.681999999999999,"15min":23.119},"min":{"1min":9.3960000000000008,"5min":8.1329999999999991,"15min":5.9169999999999998},"max":{"1min":30.167999999999999,"5min":213.73500000000001,"15min":274.91000000000003},"last":10.551},{"interface":"front","average":{"1min":21.024000000000001,"5min":32.463999999999999,"15min":32.999000000000002},"min":{"1min":14.153,"5min":11.337,"15min":11.337},"max":{"1min":31.466000000000001,"5min":234.46299999999999,"15min":339.613},"last":21.956}]}]},{"osd":2,"up_from":21,"seq":90194313656,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68040,"kb_used_data":4924,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303800,"statfs":{"total":96636764160,"available":96567091200,"internally_reserved":0,"allocated":5042176,"data_stored":2303320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:55:51 2024","interfaces":[{"interface":"back","average":{"1min":11.087,"5min":20.286999999999999,"15min":17.802},"min":{"1min":4.9820000000000002,"5min":4.9820000000000002,"15min":4.5460000000000003},"max":{"1min":23.617000000000001,"5min":335.38200000000001,"15min":335.38200000000001},"last":5.5350000000000001},{"interface":"front","average":{"1min":9.5939999999999994,"5min":20.925999999999998,"15min":17.704000000000001},"min":{"1min":5.484,"5min":5.484,"15min":4.4580000000000002},"max":{"1min":22.068999999999999,"5min":343.05599999999998,"15min":343.05599999999998},"last":7.6950000000000003}]},{"osd":1,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":16.661999999999999,"5min":32.128,"15min":25.600999999999999},"min":{"1min":7.641,"5min":5.5960000000000001,"15min":5.2190000000000003},"max":{"1min":50.104999999999997,"5min":341.34399999999999,"15min":504.58100000000002},"last":9.5600000000000005},{"interface":"front","average":{"1min":14.928000000000001,"5min":35.445,"15min":27.994},"min":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":5.7350000000000003},"max":{"1min":32.649000000000001,"5min":674.13400000000001,"15min":674.13400000000001},"last":16.646000000000001}]},{"osd":3,"last update":"Sun Jan 21 03:55:37 2024","interfaces":[{"interface":"back","average":{"1min":14.738,"5min":22.449999999999999,"15min":20.140999999999998},"min":{"1min":7.9509999999999996,"5min":6.9820000000000002,"15min":6.3659999999999997},"max":{"1min":45.518999999999998,"5min":263.31400000000002,"15min":263.31400000000002},"last":12.964},{"interface":"front","average":{"1min":17.158999999999999,"5min":26.888000000000002,"15min":22.359999999999999},"min":{"1min":9.6389999999999993,"5min":8.1180000000000003,"15min":8.1180000000000003},"max":{"1min":50.167000000000002,"5min":683.75800000000004,"15min":683.75800000000004},"last":17.747}]},{"osd":4,"last update":"Sun Jan 21 03:55:26 2024","interfaces":[{"interface":"back","average":{"1min":17.934999999999999,"5min":27.861000000000001,"15min":24.173999999999999},"min":{"1min":9.5340000000000007,"5min":8.827,"15min":8.0609999999999999},"max":{"1min":34.628,"5min":687.15800000000002,"15min":687.15800000000002},"last":10.741},{"interface":"front","average":{"1min":19.096,"5min":24.669,"15min":22.831},"min":{"1min":11.587,"5min":8.8379999999999992,"15min":8.1620000000000008},"max":{"1min":41.676000000000002,"5min":318.77199999999999,"15min":318.77199999999999},"last":18.603000000000002}]},{"osd":5,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":19.896000000000001,"5min":30.462,"15min":24.710999999999999},"min":{"1min":7.9740000000000002,"5min":7.6660000000000004,"15min":6.0650000000000004},"max":{"1min":50.735999999999997,"5min":348.47000000000003,"15min":348.47000000000003},"last":14.537000000000001},{"interface":"front","average":{"1min":21.834,"5min":30.434999999999999,"15min":25.710000000000001},"min":{"1min":10.997999999999999,"5min":10.967000000000001,"15min":8.5210000000000008},"max":{"1min":52.564999999999998,"5min":264.41199999999998,"15min":264.41199999999998},"last":11.855}]},{"osd":6,"last update":"Sun Jan 21 03:55:26 2024","interfaces":[{"interface":"back","average":{"1min":19.821000000000002,"5min":31.231999999999999,"15min":27.202000000000002},"min":{"1min":12.441000000000001,"5min":8.5960000000000001,"15min":7.2800000000000002},"max":{"1min":52.945,"5min":679.51900000000001,"15min":679.51900000000001},"last":19.686},{"interface":"front","average":{"1min":21.942,"5min":34.5,"15min":28.763999999999999},"min":{"1min":14.582000000000001,"5min":10.416,"15min":10.121},"max":{"1min":53.688000000000002,"5min":676.11800000000005,"15min":676.11800000000005},"last":20.606999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":24.338999999999999,"5min":29.204999999999998,"15min":25.914000000000001},"min":{"1min":12.472,"5min":10.278,"15min":10.278},"max":{"1min":53.186999999999998,"5min":261.48899999999998,"15min":261.48899999999998},"last":13.686999999999999},{"interface":"front","average":{"1min":24.440000000000001,"5min":38.350999999999999,"15min":27.709},"min":{"1min":15.018000000000001,"5min":12.605,"15min":8.4239999999999995},"max":{"1min":42.475000000000001,"5min":678.09500000000003,"15min":678.09500000000003},"last":21.939}]}]},{"osd":1,"up_from":21,"seq":90194313655,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94744,"kb_used_data":5316,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94277096,"statfs":{"total":96636764160,"available":96539746304,"internally_reserved":0,"allocated":5443584,"data_stored":2701160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":11.568,"5min":16.558,"15min":21.972999999999999},"min":{"1min":4.5970000000000004,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":26.126000000000001,"5min":154.52799999999999,"15min":528.255},"last":6.4720000000000004},{"interface":"front","average":{"1min":12.198,"5min":15.057,"15min":19.050999999999998},"min":{"1min":4.9619999999999997,"5min":4.5199999999999996,"15min":4.3079999999999998},"max":{"1min":24.762,"5min":127.73999999999999,"15min":447.83199999999999},"last":5.5960000000000001}]},{"osd":2,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":13.651,"5min":17.736999999999998,"15min":23.350999999999999},"min":{"1min":8.5549999999999997,"5min":7.8719999999999999,"15min":6.194},"max":{"1min":21.834,"5min":116.48999999999999,"15min":420.87700000000001},"last":11.682},{"interface":"front","average":{"1min":15.859999999999999,"5min":26.047999999999998,"15min":27.696999999999999},"min":{"1min":8.5239999999999991,"5min":7.9889999999999999,"15min":7.3209999999999997},"max":{"1min":30.548999999999999,"5min":265.24200000000002,"15min":451.637},"last":12.586}]},{"osd":3,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":20.800000000000001,"5min":22.111000000000001,"15min":23.620999999999999},"min":{"1min":9.2140000000000004,"5min":8.0370000000000008,"15min":8.0370000000000008},"max":{"1min":62.801000000000002,"5min":141.52099999999999,"15min":419.892},"last":13.31},{"interface":"front","average":{"1min":28.942,"5min":25.436,"15min":27.707000000000001},"min":{"1min":8.9109999999999996,"5min":8.6479999999999997,"15min":8.3059999999999992},"max":{"1min":123.798,"5min":143.58699999999999,"15min":425.76100000000002},"last":18.611999999999998}]},{"osd":4,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":26.920999999999999,"5min":26.507999999999999,"15min":25.122},"min":{"1min":11.009,"5min":7.4210000000000003,"15min":7.4210000000000003},"max":{"1min":106.116,"5min":229.63200000000001,"15min":344.48200000000003},"last":13.99},{"interface":"front","average":{"1min":25.748999999999999,"5min":29.904,"15min":30.959},"min":{"1min":12.282,"5min":8.3059999999999992,"15min":8.1259999999999994},"max":{"1min":65.070999999999998,"5min":236.571,"15min":385.43000000000001},"last":19.704999999999998}]},{"osd":5,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":22.766999999999999,"5min":28.076000000000001,"15min":30.946000000000002},"min":{"1min":13.015000000000001,"5min":9.5839999999999996,"15min":9.5839999999999996},"max":{"1min":38.545999999999999,"5min":138.84899999999999,"15min":338.88200000000001},"last":20.771000000000001},{"interface":"front","average":{"1min":18.858000000000001,"5min":26.266999999999999,"15min":29.963999999999999},"min":{"1min":7.577,"5min":7.577,"15min":7.577},"max":{"1min":34.542999999999999,"5min":187.03,"15min":356.423},"last":14.676}]},{"osd":6,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":21.745000000000001,"5min":31.524000000000001,"15min":31.725000000000001},"min":{"1min":10.369999999999999,"5min":10.369999999999999,"15min":9.7289999999999992},"max":{"1min":53.404000000000003,"5min":280.85000000000002,"15min":409.08100000000002},"last":15.343999999999999},{"interface":"front","average":{"1min":20.478000000000002,"5min":25.875,"15min":23.920000000000002},"min":{"1min":6.6100000000000003,"5min":6.6100000000000003,"15min":6.6100000000000003},"max":{"1min":59.128999999999998,"5min":243.083,"15min":261.75099999999998},"last":9.7629999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":21.91,"5min":28.564,"15min":25.913},"min":{"1min":12.441000000000001,"5min":11.923,"15min":7.21},"max":{"1min":43.122999999999998,"5min":169.571,"15min":357.57299999999998},"last":16.021000000000001},{"interface":"front","average":{"1min":23.428000000000001,"5min":23.902999999999999,"15min":25.715},"min":{"1min":8.6929999999999996,"5min":8.1959999999999997,"15min":7.9729999999999999},"max":{"1min":63.999000000000002,"5min":176.809,"15min":418.70600000000002},"last":10.644}]}]},{"osd":0,"up_from":21,"seq":90194313658,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72188,"kb_used_data":4912,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299652,"statfs":{"total":96636764160,"available":96562843648,"internally_reserved":0,"allocated":5029888,"data_stored":2295525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":15.691000000000001,"5min":26.472999999999999,"15min":22.228000000000002},"min":{"1min":6.7560000000000002,"5min":6.0220000000000002,"15min":5.7869999999999999},"max":{"1min":49.520000000000003,"5min":246.35400000000001,"15min":301.80599999999998},"last":9.5839999999999996},{"interface":"front","average":{"1min":21.231000000000002,"5min":29.074000000000002,"15min":24.678000000000001},"min":{"1min":11.365,"5min":6.0270000000000001,"15min":6.0270000000000001},"max":{"1min":51.290999999999997,"5min":232.851,"15min":242.80000000000001},"last":20.635999999999999}]},{"osd":2,"last update":"Sun Jan 21 03:55:44 2024","interfaces":[{"interface":"back","average":{"1min":16.657,"5min":32.222999999999999,"15min":27.050999999999998},"min":{"1min":9.2360000000000007,"5min":7.2160000000000002,"15min":6.6289999999999996},"max":{"1min":28.940000000000001,"5min":279.71300000000002,"15min":312.15800000000002},"last":14.164},{"interface":"front","average":{"1min":12.459,"5min":35.030999999999999,"15min":27.09},"min":{"1min":6.9169999999999998,"5min":6.3959999999999999,"15min":5.0910000000000002},"max":{"1min":23.015000000000001,"5min":332.57499999999999,"15min":357.38900000000001},"last":11.573}]},{"osd":3,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":22.167999999999999,"5min":43.600000000000001,"15min":32.090000000000003},"min":{"1min":7.5049999999999999,"5min":7.5049999999999999,"15min":6.9829999999999997},"max":{"1min":68.963999999999999,"5min":547.15800000000002,"15min":547.15800000000002},"last":21.332000000000001},{"interface":"front","average":{"1min":20.045999999999999,"5min":44.688000000000002,"15min":32.734999999999999},"min":{"1min":10.461,"5min":7.5979999999999999,"15min":7.5979999999999999},"max":{"1min":74.480999999999995,"5min":361.11500000000001,"15min":361.11500000000001},"last":10.859999999999999}]},{"osd":4,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":22.556000000000001,"5min":35.131,"15min":25.462},"min":{"1min":10.24,"5min":7.4630000000000001,"15min":7.3019999999999996},"max":{"1min":75.359999999999999,"5min":333.23899999999998,"15min":333.23899999999998},"last":13.468},{"interface":"front","average":{"1min":27.826000000000001,"5min":41.021000000000001,"15min":28.832999999999998},"min":{"1min":14.176,"5min":10.836,"15min":6.3630000000000004},"max":{"1min":90.658000000000001,"5min":333.87400000000002,"15min":333.87400000000002},"last":12.227}]},{"osd":5,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":12.843,"5min":40.453000000000003,"15min":31.494},"min":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"max":{"1min":66.775999999999996,"5min":387.58100000000002,"15min":543.84100000000001},"last":7.1559999999999997},{"interface":"front","average":{"1min":26.053999999999998,"5min":30.795999999999999,"15min":23.568000000000001},"min":{"1min":10.856999999999999,"5min":6.3529999999999998,"15min":5.452},"max":{"1min":88.313999999999993,"5min":266.03800000000001,"15min":335.589},"last":17.152000000000001}]},{"osd":6,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":16.995999999999999,"5min":31.850000000000001,"15min":26.893999999999998},"min":{"1min":9.0709999999999997,"5min":6.1059999999999999,"15min":6.1059999999999999},"max":{"1min":67.790999999999997,"5min":311.358,"15min":370.30200000000002},"last":14.861000000000001},{"interface":"front","average":{"1min":25.103000000000002,"5min":33.008000000000003,"15min":26.288},"min":{"1min":12.054,"5min":8.2769999999999992,"15min":7.2389999999999999},"max":{"1min":70.632000000000005,"5min":230.27699999999999,"15min":230.27699999999999},"last":24.247}]},{"osd":7,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":24.888000000000002,"5min":30.061,"15min":25.626000000000001},"min":{"1min":10.714,"5min":9.1110000000000007,"15min":8.8960000000000008},"max":{"1min":71.314999999999998,"5min":131.65000000000001,"15min":214.071},"last":23.248999999999999},{"interface":"front","average":{"1min":28.067,"5min":38.735999999999997,"15min":28.626000000000001},"min":{"1min":12.391,"5min":8.8070000000000004,"15min":8.8070000000000004},"max":{"1min":94.670000000000002,"5min":348.38600000000002,"15min":348.38600000000002},"last":18.812000000000001}]}]},{"osd":6,"up_from":18,"seq":77309411770,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72208,"kb_used_data":4928,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299632,"statfs":{"total":96636764160,"available":96562823168,"internally_reserved":0,"allocated":5046272,"data_stored":2303325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":16.803000000000001,"5min":22.076000000000001,"15min":21.289999999999999},"min":{"1min":5.3380000000000001,"5min":5.3380000000000001,"15min":4.391},"max":{"1min":98.180000000000007,"5min":284.42500000000001,"15min":438.98899999999998},"last":12.811},{"interface":"front","average":{"1min":28.359999999999999,"5min":27.478999999999999,"15min":21.431000000000001},"min":{"1min":8.7149999999999999,"5min":5.2160000000000002,"15min":4.8840000000000003},"max":{"1min":144.92400000000001,"5min":380.07600000000002,"15min":380.07600000000002},"last":10.606999999999999}]},{"osd":1,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":19.898,"5min":31.847999999999999,"15min":24.283000000000001},"min":{"1min":8.7289999999999992,"5min":6.883,"15min":4.7389999999999999},"max":{"1min":54.722999999999999,"5min":526.48099999999999,"15min":526.48099999999999},"last":7.7670000000000003},{"interface":"front","average":{"1min":14.859999999999999,"5min":26.492000000000001,"15min":22.867999999999999},"min":{"1min":6.3600000000000003,"5min":4.3120000000000003,"15min":4.3120000000000003},"max":{"1min":44.970999999999997,"5min":395.72399999999999,"15min":395.72399999999999},"last":9.3170000000000002}]},{"osd":2,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":12.648,"5min":21.052,"15min":20.300999999999998},"min":{"1min":6.0350000000000001,"5min":5.8920000000000003,"15min":5.8920000000000003},"max":{"1min":35.268999999999998,"5min":209.78,"15min":213.83199999999999},"last":12.169},{"interface":"front","average":{"1min":22.007000000000001,"5min":31.407,"15min":26.654},"min":{"1min":6.7210000000000001,"5min":6.7210000000000001,"15min":6.7210000000000001},"max":{"1min":77.159000000000006,"5min":444.822,"15min":444.822},"last":26.718}]},{"osd":3,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":18.542999999999999,"5min":33.561,"15min":26.021000000000001},"min":{"1min":8.1959999999999997,"5min":6.1890000000000001,"15min":6.1890000000000001},"max":{"1min":59.645000000000003,"5min":527.40899999999999,"15min":527.40899999999999},"last":27.533000000000001},{"interface":"front","average":{"1min":17.805,"5min":25.48,"15min":23.213999999999999},"min":{"1min":6.4089999999999998,"5min":6.4050000000000002,"15min":6.4050000000000002},"max":{"1min":57.729999999999997,"5min":370.10000000000002,"15min":370.10000000000002},"last":14.664999999999999}]},{"osd":4,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":21.780000000000001,"5min":31.358000000000001,"15min":24.521999999999998},"min":{"1min":6.9160000000000004,"5min":6.3120000000000003,"15min":6.3120000000000003},"max":{"1min":95.962000000000003,"5min":385.49200000000002,"15min":385.49200000000002},"last":6.5410000000000004},{"interface":"front","average":{"1min":21.581,"5min":26.248000000000001,"15min":21.954000000000001},"min":{"1min":8.0660000000000007,"5min":7.2130000000000001,"15min":6.7539999999999996},"max":{"1min":89.241,"5min":241.59200000000001,"15min":241.59200000000001},"last":9.9629999999999992}]},{"osd":5,"last update":"Sun Jan 21 03:55:30 2024","interfaces":[{"interface":"back","average":{"1min":20.192,"5min":27.117000000000001,"15min":25.914000000000001},"min":{"1min":8.4700000000000006,"5min":8.2810000000000006,"15min":7.5679999999999996},"max":{"1min":63.576999999999998,"5min":488.01299999999998,"15min":488.01299999999998},"last":25.760000000000002},{"interface":"front","average":{"1min":18.07,"5min":21.914999999999999,"15min":26.193999999999999},"min":{"1min":9.5719999999999992,"5min":8.1890000000000001,"15min":8.1890000000000001},"max":{"1min":32.356000000000002,"5min":197.38,"15min":333.43900000000002},"last":13.863}]},{"osd":7,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":23.504000000000001,"5min":30.591999999999999,"15min":25.545000000000002},"min":{"1min":10.676,"5min":7.1340000000000003,"15min":7.1340000000000003},"max":{"1min":100.84399999999999,"5min":322.54599999999999,"15min":322.54599999999999},"last":8.3339999999999996},{"interface":"front","average":{"1min":24.988,"5min":29.010999999999999,"15min":25.050000000000001},"min":{"1min":13.025,"5min":9.6940000000000008,"15min":8.6699999999999999},"max":{"1min":90.594999999999999,"5min":218.30199999999999,"15min":218.30199999999999},"last":15.365}]}]},{"osd":5,"up_from":18,"seq":77309411773,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72600,"kb_used_data":5316,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299240,"statfs":{"total":96636764160,"available":96562421760,"internally_reserved":0,"allocated":5443584,"data_stored":2701160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":9.1660000000000004,"5min":22.483000000000001,"15min":20.535},"min":{"1min":4.9420000000000002,"5min":4.6539999999999999,"15min":4.0940000000000003},"max":{"1min":28.151,"5min":241.49000000000001,"15min":346.541},"last":7.2160000000000002},{"interface":"front","average":{"1min":13.221,"5min":26.934000000000001,"15min":26.605},"min":{"1min":7.6550000000000002,"5min":6.1500000000000004,"15min":5.8819999999999997},"max":{"1min":24.053000000000001,"5min":236.81100000000001,"15min":524.904},"last":12.234999999999999}]},{"osd":1,"last update":"Sun Jan 21 03:55:22 2024","interfaces":[{"interface":"back","average":{"1min":14.785,"5min":26.623999999999999,"15min":20.832999999999998},"min":{"1min":6.2729999999999997,"5min":5.1429999999999998,"15min":4.907},"max":{"1min":84.840999999999994,"5min":284.07400000000001,"15min":368.12400000000002},"last":8.4740000000000002},{"interface":"front","average":{"1min":20.699999999999999,"5min":34.825000000000003,"15min":27.59},"min":{"1min":11.284000000000001,"5min":8.8569999999999993,"15min":7.5359999999999996},"max":{"1min":85.671000000000006,"5min":296.56400000000002,"15min":368.86099999999999},"last":10.801}]},{"osd":2,"last update":"Sun Jan 21 03:55:35 2024","interfaces":[{"interface":"back","average":{"1min":11.273999999999999,"5min":46.090000000000003,"15min":29.077000000000002},"min":{"1min":7.25,"5min":6.6310000000000002,"15min":6.0579999999999998},"max":{"1min":27.489000000000001,"5min":2166.2739999999999,"15min":2166.2739999999999},"last":10.039999999999999},{"interface":"front","average":{"1min":19.923999999999999,"5min":53.329000000000001,"15min":37.494},"min":{"1min":9.8030000000000008,"5min":7.1150000000000002,"15min":7.1150000000000002},"max":{"1min":71.177999999999997,"5min":2181.6190000000001,"15min":2181.6190000000001},"last":15.766}]},{"osd":3,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":16.638000000000002,"5min":30.282,"15min":26.837},"min":{"1min":6.8179999999999996,"5min":6.2770000000000001,"15min":6.2770000000000001},"max":{"1min":63.612000000000002,"5min":412.714,"15min":412.714},"last":16.672999999999998},{"interface":"front","average":{"1min":20.707999999999998,"5min":34.683999999999997,"15min":28.085999999999999},"min":{"1min":9.9399999999999995,"5min":9.6389999999999993,"15min":6.8840000000000003},"max":{"1min":57.048000000000002,"5min":314.28199999999998,"15min":314.28199999999998},"last":13.682}]},{"osd":4,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":21.478000000000002,"5min":39.764000000000003,"15min":27.739999999999998},"min":{"1min":12.226000000000001,"5min":7.7060000000000004,"15min":6.274},"max":{"1min":49.774999999999999,"5min":418.18299999999999,"15min":418.18299999999999},"last":18.27},{"interface":"front","average":{"1min":24.219000000000001,"5min":42.651000000000003,"15min":30.201000000000001},"min":{"1min":14.734999999999999,"5min":11.993,"15min":7.5490000000000004},"max":{"1min":57.712000000000003,"5min":387.125,"15min":387.125},"last":19.117999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":17.492999999999999,"5min":32.762,"15min":25.588999999999999},"min":{"1min":8.6739999999999995,"5min":6.7050000000000001,"15min":6.7050000000000001},"max":{"1min":31.645,"5min":431.40899999999999,"15min":431.40899999999999},"last":11.577},{"interface":"front","average":{"1min":23.259,"5min":36.051000000000002,"15min":28.965},"min":{"1min":11.241,"5min":9.0809999999999995,"15min":8.5470000000000006},"max":{"1min":61.225999999999999,"5min":432.19,"15min":432.19},"last":13.022}]},{"osd":7,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":23.452000000000002,"5min":38.652000000000001,"15min":29.713000000000001},"min":{"1min":13.459,"5min":9.7560000000000002,"15min":7.8639999999999999},"max":{"1min":48.789999999999999,"5min":281.06099999999998,"15min":281.06099999999998},"last":17.498999999999999},{"interface":"front","average":{"1min":12.581,"5min":26.823,"15min":22.974},"min":{"1min":6.5170000000000003,"5min":6.5170000000000003,"15min":6.5170000000000003},"max":{"1min":36.563000000000002,"5min":282.29000000000002,"15min":282.29000000000002},"last":9.2140000000000004}]}]},{"osd":7,"up_from":14,"seq":60129542586,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72224,"kb_used_data":4944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299616,"statfs":{"total":96636764160,"available":96562806784,"internally_reserved":0,"allocated":5062656,"data_stored":2314831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:17 2024","interfaces":[{"interface":"back","average":{"1min":11.395,"5min":20.713000000000001,"15min":16.225999999999999},"min":{"1min":5.0099999999999998,"5min":4.7450000000000001,"15min":4.1929999999999996},"max":{"1min":27.079000000000001,"5min":239.37799999999999,"15min":304.20800000000003},"last":26.015999999999998},{"interface":"front","average":{"1min":10.247999999999999,"5min":21.539000000000001,"15min":16.581},"min":{"1min":5.9039999999999999,"5min":5.2709999999999999,"15min":4.2679999999999998},"max":{"1min":31.059999999999999,"5min":181.04599999999999,"15min":181.04599999999999},"last":22.859000000000002}]},{"osd":1,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":12.342000000000001,"5min":19.533000000000001,"15min":20.614000000000001},"min":{"1min":6.0869999999999997,"5min":5.6580000000000004,"15min":5.0419999999999998},"max":{"1min":27.402999999999999,"5min":138.82499999999999,"15min":248.17500000000001},"last":13.691000000000001},{"interface":"front","average":{"1min":14.029,"5min":23.824999999999999,"15min":23.670999999999999},"min":{"1min":7.9109999999999996,"5min":6.3769999999999998,"15min":6.3769999999999998},"max":{"1min":26.454000000000001,"5min":214.55000000000001,"15min":277.85700000000003},"last":11.555}]},{"osd":2,"last update":"Sun Jan 21 03:56:17 2024","interfaces":[{"interface":"back","average":{"1min":17.594999999999999,"5min":36.784999999999997,"15min":27.683},"min":{"1min":7.2869999999999999,"5min":7.2290000000000001,"15min":6.1779999999999999},"max":{"1min":47.972999999999999,"5min":683.21000000000004,"15min":683.21000000000004},"last":18.545000000000002},{"interface":"front","average":{"1min":16.43,"5min":24.434999999999999,"15min":22.135999999999999},"min":{"1min":9.0399999999999991,"5min":6.3890000000000002,"15min":6.3890000000000002},"max":{"1min":35.869,"5min":150.369,"15min":367.30000000000001},"last":15.68}]},{"osd":3,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":17.163,"5min":27.100000000000001,"15min":22.632000000000001},"min":{"1min":10.734999999999999,"5min":7.1470000000000002,"15min":6.8040000000000003},"max":{"1min":24.895,"5min":192.14599999999999,"15min":192.14599999999999},"last":12.595000000000001},{"interface":"front","average":{"1min":16.158000000000001,"5min":24.597000000000001,"15min":21.649000000000001},"min":{"1min":7.851,"5min":7.8090000000000002,"15min":6.5960000000000001},"max":{"1min":29.548999999999999,"5min":166.846,"15min":210.006},"last":9.8450000000000006}]},{"osd":4,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":20.638999999999999,"5min":27.516999999999999,"15min":22.658999999999999},"min":{"1min":9.4700000000000006,"5min":7.6260000000000003,"15min":7.2309999999999999},"max":{"1min":43.503,"5min":197.31,"15min":268.416},"last":30.398},{"interface":"front","average":{"1min":17.760999999999999,"5min":25.548999999999999,"15min":21.73},"min":{"1min":7.516,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":33.429000000000002,"5min":143.07400000000001,"15min":168.95599999999999},"last":28.981999999999999}]},{"osd":5,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":17.228999999999999,"5min":25.649999999999999,"15min":24.148},"min":{"1min":7.3029999999999999,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":25.670000000000002,"5min":149.56,"15min":226.947},"last":20.795999999999999},{"interface":"front","average":{"1min":17.081,"5min":24.050000000000001,"15min":23.798999999999999},"min":{"1min":8.6419999999999995,"5min":7.5839999999999996,"15min":7.5839999999999996},"max":{"1min":41.097000000000001,"5min":137.00200000000001,"15min":248.94800000000001},"last":28.298999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":21.542000000000002,"5min":27.488,"15min":26.065999999999999},"min":{"1min":10.897,"5min":9.4459999999999997,"15min":9.4459999999999997},"max":{"1min":55.081000000000003,"5min":193.16,"15min":295.12099999999998},"last":27.280000000000001},{"interface":"front","average":{"1min":19.73,"5min":28.16,"15min":26.513999999999999},"min":{"1min":11.029,"5min":9.3219999999999992,"15min":9.3219999999999992},"max":{"1min":55.758000000000003,"5min":178.01900000000001,"15min":278.71899999999999},"last":23.907}]}]},{"osd":4,"up_from":14,"seq":60129542585,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72220,"kb_used_data":4936,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299620,"statfs":{"total":96636764160,"available":96562810880,"internally_reserved":0,"allocated":5054464,"data_stored":2310730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":8.7889999999999997,"5min":15.209,"15min":16.693000000000001},"min":{"1min":4.9939999999999998,"5min":4.4660000000000002,"15min":4.4100000000000001},"max":{"1min":18.459,"5min":220.72900000000001,"15min":220.72900000000001},"last":5.0410000000000004},{"interface":"front","average":{"1min":12.965,"5min":16.300000000000001,"15min":18.149999999999999},"min":{"1min":6.6360000000000001,"5min":4.6029999999999998,"15min":4.6029999999999998},"max":{"1min":45.616,"5min":101.768,"15min":204.142},"last":23.486999999999998}]},{"osd":1,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":12.475,"5min":22.411999999999999,"15min":18.585000000000001},"min":{"1min":6.6440000000000001,"5min":4.9359999999999999,"15min":4.9359999999999999},"max":{"1min":26.292999999999999,"5min":272.61399999999998,"15min":272.61399999999998},"last":20.783999999999999},{"interface":"front","average":{"1min":16.969000000000001,"5min":24.193000000000001,"15min":20.829000000000001},"min":{"1min":9.1150000000000002,"5min":6.8719999999999999,"15min":5.6200000000000001},"max":{"1min":49.414999999999999,"5min":142.97300000000001,"15min":223.49299999999999},"last":34.969000000000001}]},{"osd":2,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":23.027000000000001,"5min":29.632000000000001,"15min":24.187999999999999},"min":{"1min":9.3979999999999997,"5min":7.4320000000000004,"15min":6.6879999999999997},"max":{"1min":77.858999999999995,"5min":285.61000000000001,"15min":285.61000000000001},"last":25.93},{"interface":"front","average":{"1min":21.07,"5min":26.555,"15min":22.710999999999999},"min":{"1min":7.6769999999999996,"5min":6.899,"15min":6.2080000000000002},"max":{"1min":101.676,"5min":298.892,"15min":298.892},"last":8.9760000000000009}]},{"osd":3,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":14.119,"5min":19.032,"15min":20.623999999999999},"min":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"max":{"1min":48.497,"5min":132.958,"15min":200.697},"last":29.013999999999999},{"interface":"front","average":{"1min":19.265000000000001,"5min":23.931999999999999,"15min":21.149000000000001},"min":{"1min":8.9890000000000008,"5min":7.5250000000000004,"15min":7.1539999999999999},"max":{"1min":60.786000000000001,"5min":166.66200000000001,"15min":188.06399999999999},"last":24.388000000000002}]},{"osd":5,"last update":"Sun Jan 21 03:55:58 2024","interfaces":[{"interface":"back","average":{"1min":17.315000000000001,"5min":26.736000000000001,"15min":24.969000000000001},"min":{"1min":10.885999999999999,"5min":8.2970000000000006,"15min":6.7359999999999998},"max":{"1min":30.718,"5min":182.53700000000001,"15min":470.87700000000001},"last":19.791},{"interface":"front","average":{"1min":13.831,"5min":23.012,"15min":24.087},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":6.944},"max":{"1min":23.405000000000001,"5min":145.125,"15min":471.56999999999999},"last":26.879999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:55:37 2024","interfaces":[{"interface":"back","average":{"1min":21.370999999999999,"5min":25.059999999999999,"15min":21.741},"min":{"1min":14.724,"5min":8.5820000000000007,"15min":6.415},"max":{"1min":73.459999999999994,"5min":208.42099999999999,"15min":285.87},"last":30.898},{"interface":"front","average":{"1min":18.635000000000002,"5min":26.960000000000001,"15min":24.614000000000001},"min":{"1min":9.5129999999999999,"5min":9.5129999999999999,"15min":7.5430000000000001},"max":{"1min":64.510000000000005,"5min":252.47999999999999,"15min":252.47999999999999},"last":25.187999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:55:58 2024","interfaces":[{"interface":"back","average":{"1min":20.036999999999999,"5min":25.931000000000001,"15min":22.280000000000001},"min":{"1min":12.244,"5min":9.3040000000000003,"15min":7.1820000000000004},"max":{"1min":44.164999999999999,"5min":141.51300000000001,"15min":229.04900000000001},"last":28.405000000000001},{"interface":"front","average":{"1min":22.382999999999999,"5min":28.206,"15min":22.698},"min":{"1min":15.759,"5min":12.417999999999999,"15min":7.4299999999999997},"max":{"1min":42.031999999999996,"5min":179.12799999999999,"15min":179.12799999999999},"last":31.728000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T03:56:26.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T03:56:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.342+0000 7f8838c34640 10 monclient: get_monmap_and_config 2024-01-21T03:56:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.342+0000 7f8838c34640 10 monclient: build_initial_monmap 2024-01-21T03:56:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.342+0000 7f8838c34640 10 monclient: monmap: 2024-01-21T03:56:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:26.343414+0000 2024-01-21T03:56:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:26.343414+0000 2024-01-21T03:56:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:26.345 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 1 Processor -- start 2024-01-21T03:56:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 1 -- start start 2024-01-21T03:56:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 10 monclient: init 2024-01-21T03:56:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-01-21T03:56:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88340a8a40 0x7f8834145b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 10 monclient(hunting): picked mon.noname-a con 0x7f88340a8a40 addr v2:172.21.15.7:3300/0 2024-01-21T03:56:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88341460a0 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f883413f8b0 0x7f883414c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 10 monclient(hunting): picked mon.noname-b con 0x7f883413f8b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88340aa9b0 con 0x7f88341460a0 2024-01-21T03:56:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88340aa7d0 con 0x7f88340a8a40 2024-01-21T03:56:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88340aac90 con 0x7f883413f8b0 2024-01-21T03:56:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.343+0000 7f8838c34640 10 monclient(hunting): authenticate will time out at 3532.659585s 2024-01-21T03:56:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.344+0000 7f883259c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88340a8a40 0x7f8834145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.344+0000 7f883259c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88340a8a40 0x7f8834145b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42024/0 (socket says 172.21.15.7:42024) 2024-01-21T03:56:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.344+0000 7f883259c640 1 -- 172.21.15.7:0/2091690850 learned_addr learned my addr 172.21.15.7:0/2091690850 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.344+0000 7f883259c640 10 monclient(hunting): get_auth_request con 0x7f88340a8a40 auth_method 0 2024-01-21T03:56:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.344+0000 7f883259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.344+0000 7f883259c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.344+0000 7f883259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.345+0000 7f883259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.345+0000 7f883259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.345+0000 7f883259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.346+0000 7f883259c640 10 monclient(hunting): handle_auth_done global_id 15825 payload 274 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.346+0000 7f883259c640 1 -- 172.21.15.7:0/2091690850 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f883413f8b0 msgr2=0x7f883414c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.346+0000 7f883259c640 1 --2- 172.21.15.7:0/2091690850 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f883413f8b0 0x7f883414c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.346+0000 7f883259c640 1 -- 172.21.15.7:0/2091690850 >> v1:172.21.15.177:6789/0 conn(0x7f88341460a0 legacy=0x7f883413f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.346+0000 7f883259c640 10 monclient: _finish_hunting 0 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.346+0000 7f883259c640 1 monclient: found mon.noname-a 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.346+0000 7f883259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.346+0000 7f883259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.346+0000 7f883259c640 1 -- 172.21.15.7:0/2091690850 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88340aaf70 con 0x7f88340a8a40 2024-01-21T03:56:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.348+0000 7f883259c640 1 --2- 172.21.15.7:0/2091690850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88340a8a40 0x7f8834145b60 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7f882c009e50 tx=0x7f882c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8e5b0e45c290d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f883159a640 1 -- 172.21.15.7:0/2091690850 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c02cb20 con 0x7f88340a8a40 2024-01-21T03:56:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f883159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f883159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f883159a640 10 monclient: dump: 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f883159a640 10 monclient: _finish_auth 0 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f883159a640 10 monclient: _check_auth_tickets 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f883159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f883159a640 1 -- 172.21.15.7:0/2091690850 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882c02cc80 con 0x7f88340a8a40 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f8838c34640 5 monclient: authenticate success, global_id 15825 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f8838c34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f883159a640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f883159a640 1 -- 172.21.15.7:0/2091690850 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c0343f0 con 0x7f88340a8a40 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f8838c34640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f8838c34640 4 set_mon_vals no callback set 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f883159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f883159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f883159a640 10 monclient: dump: 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:26.356 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:26.356 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:26.356 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:26.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f8838c34640 10 monclient: shutdown 2024-01-21T03:56:26.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f8838c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:26.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f8838c34640 1 -- 172.21.15.7:0/2091690850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88340a8a40 msgr2=0x7f8834145b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.353+0000 7f8838c34640 1 --2- 172.21.15.7:0/2091690850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88340a8a40 0x7f8834145b60 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7f882c009e50 tx=0x7f882c02bf80 comp rx=0 tx=0).stop 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.354+0000 7f8838c34640 1 -- 172.21.15.7:0/2091690850 shutdown_connections 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.354+0000 7f8838c34640 1 --2- 172.21.15.7:0/2091690850 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f883413f8b0 0x7f883414c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.354+0000 7f8838c34640 1 --2- 172.21.15.7:0/2091690850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88340a8a40 0x7f8834145b60 unknown :-1 s=CLOSED pgs=3083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.354+0000 7f8838c34640 1 -- 172.21.15.7:0/2091690850 >> 172.21.15.7:0/2091690850 conn(0x7f8834066f50 msgr2=0x7f88340b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.354+0000 7f8838c34640 1 -- 172.21.15.7:0/2091690850 shutdown_connections 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.354+0000 7f8838c34640 1 -- 172.21.15.7:0/2091690850 wait complete. 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.355+0000 7f8838c34640 10 monclient: build_initial_monmap 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.355+0000 7f8838c34640 10 monclient: monmap: 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:26.363 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.355+0000 7f8838c34640 1 librados: starting msgr at 2024-01-21T03:56:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.355+0000 7f8838c34640 1 librados: starting objecter 2024-01-21T03:56:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.355+0000 7f8838c34640 1 Processor -- start 2024-01-21T03:56:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.355+0000 7f8838c34640 1 -- start start 2024-01-21T03:56:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.355+0000 7f8838c34640 1 librados: setting wanted keys 2024-01-21T03:56:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.355+0000 7f8838c34640 1 librados: calling monclient init 2024-01-21T03:56:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.355+0000 7f8838c34640 10 monclient: init 2024-01-21T03:56:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T03:56:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88340a8a40 0x7f88340b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 10 monclient(hunting): picked mon.noname-b con 0x7f88340a8a40 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f883413f8b0 0x7f8834142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 10 monclient(hunting): picked mon.noname-a con 0x7f883413f8b0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f88341460a0 0x7f88341d7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 10 monclient(hunting): picked mon.noname-c con 0x7f88341460a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88341d7ca0 con 0x7f883413f8b0 2024-01-21T03:56:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88341d80c0 con 0x7f88340a8a40 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88341d8280 con 0x7f88341460a0 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8838c34640 10 monclient(hunting): authenticate will time out at 3532.671950s 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.356+0000 7f8831d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f883413f8b0 0x7f8834142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.357+0000 7f8831d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f883413f8b0 0x7f8834142350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42028/0 (socket says 172.21.15.7:42028) 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.357+0000 7f8831d9b640 1 -- 172.21.15.7:0/2963177297 learned_addr learned my addr 172.21.15.7:0/2963177297 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.357+0000 7f8831d9b640 10 monclient(hunting): get_auth_request con 0x7f883413f8b0 auth_method 0 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.357+0000 7f8831d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.357+0000 7f8831d9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.357+0000 7f8831d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.357+0000 7f8831d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.357+0000 7f8831d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.357+0000 7f8831d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.360+0000 7f8831d9b640 10 monclient(hunting): handle_auth_done global_id 15828 payload 824 2024-01-21T03:56:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.360+0000 7f8831d9b640 1 -- 172.21.15.7:0/2963177297 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f88341460a0 msgr2=0x7f88341d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.360+0000 7f8831d9b640 1 --2- 172.21.15.7:0/2963177297 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f88341460a0 0x7f88341d7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.360+0000 7f8831d9b640 1 -- 172.21.15.7:0/2963177297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88340a8a40 msgr2=0x7f88340b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.360+0000 7f8831d9b640 1 --2- 172.21.15.7:0/2963177297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88340a8a40 0x7f88340b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.360+0000 7f8831d9b640 10 monclient: _finish_hunting 0 2024-01-21T03:56:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.360+0000 7f8831d9b640 1 monclient: found mon.noname-a 2024-01-21T03:56:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.360+0000 7f8831d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.360+0000 7f8831d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:56:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.360+0000 7f8831d9b640 1 -- 172.21.15.7:0/2963177297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c009b30 con 0x7f883413f8b0 2024-01-21T03:56:26.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.362+0000 7f8831d9b640 1 --2- 172.21.15.7:0/2963177297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f883413f8b0 0x7f8834142350 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7f881c00b3e0 tx=0x7f881c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:26.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.362+0000 7f882affd640 10 client.?.objecter ms_handle_connect 0x7f883413f8b0 2024-01-21T03:56:26.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.362+0000 7f882affd640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:26.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f882affd640 1 -- 172.21.15.7:0/2963177297 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c011830 con 0x7f883413f8b0 2024-01-21T03:56:26.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f882affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:26.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f882affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:56:26.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f882affd640 10 monclient: dump: 2024-01-21T03:56:26.368 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:26.368 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:26.368 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:26.368 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:26.368 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:26.368 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:26.368 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:26.368 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f882affd640 10 monclient: _finish_auth 0 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f882affd640 10 monclient: _check_auth_tickets 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f882affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f882affd640 1 -- 172.21.15.7:0/2963177297 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881c0119b0 con 0x7f883413f8b0 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f8838c34640 5 monclient: authenticate success, global_id 15828 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f8838c34640 10 monclient: _renew_subs 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f8838c34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f8838c34640 1 -- 172.21.15.7:0/2963177297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88340b53b0 con 0x7f883413f8b0 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f882affd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f882affd640 1 -- 172.21.15.7:0/2963177297 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c011d60 con 0x7f883413f8b0 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f882affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f882affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:56:26.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f882affd640 10 monclient: dump: 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f8830d99640 4 set_mon_vals no callback set 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f8838c34640 10 client.15828.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f8838c34640 10 monclient: _renew_subs 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f8838c34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f8838c34640 1 -- 172.21.15.7:0/2963177297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88340b58c0 con 0x7f883413f8b0 2024-01-21T03:56:26.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.367+0000 7f8838c34640 1 librados: init done 2024-01-21T03:56:26.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.368+0000 7f8838c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:26.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.368+0000 7f8838c34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:26.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.368+0000 7f8838c34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:26.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.368+0000 7f8838c34640 1 -- 172.21.15.7:0/2963177297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88340a9cb0 con 0x7f883413f8b0 2024-01-21T03:56:26.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.392+0000 7f882affd640 1 -- 172.21.15.7:0/2963177297 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f881c01a400 con 0x7f883413f8b0 2024-01-21T03:56:26.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.393+0000 7f882affd640 1 --2- 172.21.15.7:0/2963177297 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f881406c860 0x7f881406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:26.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.393+0000 7f883259c640 1 --2- 172.21.15.7:0/2963177297 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f881406c860 0x7f881406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:26.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.393+0000 7f883259c640 10 monclient: get_auth_request con 0x7f881406c860 auth_method 0 2024-01-21T03:56:26.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.393+0000 7f882affd640 10 client.15828.objecter ms_handle_connect 0x7f881406c860 2024-01-21T03:56:26.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.393+0000 7f883259c640 1 --2- 172.21.15.7:0/2963177297 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f881406c860 0x7f881406ed20 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f882c035000 tx=0x7f882c02e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:26.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.643+0000 7f882affd640 1 -- 172.21.15.7:0/2963177297 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7f881c058a20 con 0x7f883413f8b0 2024-01-21T03:56:26.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.643+0000 7f882affd640 10 client.15828.objecter ms_dispatch 0x7f8834000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:26.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.643+0000 7f882affd640 3 client.15828.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:26.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.643+0000 7f882affd640 3 client.15828.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.643+0000 7f882affd640 20 client.15828.objecter dump_active .. 0 homeless 2024-01-21T03:56:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.643+0000 7f882affd640 1 -- 172.21.15.7:0/2963177297 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f881c054d60 con 0x7f883413f8b0 2024-01-21T03:56:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.643+0000 7f882affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.643+0000 7f882affd640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:26.720 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:41.035 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:26.723 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:41.038 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:26.737 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:41.052 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:26.765 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:41.025 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:26.768 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:41.028 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:26.781 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:41.041 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:26.789 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:41.104 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:26.794 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:41.109 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:26.811 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:41.126 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:26.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.841+0000 7f8838c34640 1 -- 172.21.15.7:0/2963177297 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88340b5b40 con 0x7f881406c860 2024-01-21T03:56:26.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.846+0000 7f882affd640 1 -- 172.21.15.7:0/2963177297 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151878 (secure 0 0 0) 0x7f88340b5b40 con 0x7f881406c860 2024-01-21T03:56:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:56:26.866 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-01-21T03:56:26.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 10 librados: watch_flush enter 2024-01-21T03:56:26.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 10 librados: watch_flush exit 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 20 client.15828.objecter shutdown clearing up homeless session... 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 10 client.15828.objecter successfully canceled tick 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 1 -- 172.21.15.7:0/2963177297 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f881406c860 msgr2=0x7f881406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 1 --2- 172.21.15.7:0/2963177297 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f881406c860 0x7f881406ed20 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f882c035000 tx=0x7f882c02e000 comp rx=0 tx=0).stop 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 10 monclient: shutdown 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 1 -- 172.21.15.7:0/2963177297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f883413f8b0 msgr2=0x7f8834142350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 1 --2- 172.21.15.7:0/2963177297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f883413f8b0 0x7f8834142350 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7f881c00b3e0 tx=0x7f881c00b8a0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 1 -- 172.21.15.7:0/2963177297 shutdown_connections 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 1 --2- 172.21.15.7:0/2963177297 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f881406c860 0x7f881406ed20 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 1 --2- 172.21.15.7:0/2963177297 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f88341460a0 0x7f88341d7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 1 --2- 172.21.15.7:0/2963177297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f883413f8b0 0x7f8834142350 unknown :-1 s=CLOSED pgs=3084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 1 --2- 172.21.15.7:0/2963177297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88340a8a40 0x7f88340b0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 1 -- 172.21.15.7:0/2963177297 >> 172.21.15.7:0/2963177297 conn(0x7f8834066f50 msgr2=0x7f88340b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:26.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 1 -- 172.21.15.7:0/2963177297 shutdown_connections 2024-01-21T03:56:26.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 1 -- 172.21.15.7:0/2963177297 wait complete. 2024-01-21T03:56:26.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.849+0000 7f8838c34640 1 librados: shutdown 2024-01-21T03:56:26.870 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":2398,"stamp":"2024-01-21T03:56:26.232835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":600688,"kb_used_data":40292,"kb_used_omap":12,"kb_used_meta":560243,"kb_avail":754374032,"statfs":{"total":773094113280,"available":772479008768,"internally_reserved":0,"allocated":41259008,"data_stored":19284616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":573689393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":146,"apply_latency_ms":146,"commit_latency_ns":146000000,"apply_latency_ns":146000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.330366"},"pg_stats":[{"pgid":"14.1e","version":"0'0","reported_seq":1254,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.191591+0000","last_change":"2024-01-21T03:21:15.540562+0000","last_active":"2024-01-21T03:56:23.191591+0000","last_peered":"2024-01-21T03:56:23.191591+0000","last_clean":"2024-01-21T03:56:23.191591+0000","last_became_active":"2024-01-21T03:21:15.289479+0000","last_became_peered":"2024-01-21T03:21:15.289479+0000","last_unstale":"2024-01-21T03:56:23.191591+0000","last_undegraded":"2024-01-21T03:56:23.191591+0000","last_fullsized":"2024-01-21T03:56:23.191591+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:26:47.504093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1210,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.683250+0000","last_change":"2024-01-21T03:21:15.879415+0000","last_active":"2024-01-21T03:56:20.683250+0000","last_peered":"2024-01-21T03:56:20.683250+0000","last_clean":"2024-01-21T03:56:20.683250+0000","last_became_active":"2024-01-21T03:21:15.864718+0000","last_became_peered":"2024-01-21T03:21:15.864718+0000","last_unstale":"2024-01-21T03:56:20.683250+0000","last_undegraded":"2024-01-21T03:56:20.683250+0000","last_fullsized":"2024-01-21T03:56:20.683250+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:26:29.398180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1231,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.706988+0000","last_change":"2024-01-21T03:21:09.350719+0000","last_active":"2024-01-21T03:56:19.706988+0000","last_peered":"2024-01-21T03:56:19.706988+0000","last_clean":"2024-01-21T03:56:19.706988+0000","last_became_active":"2024-01-21T03:21:09.333873+0000","last_became_peered":"2024-01-21T03:21:09.333873+0000","last_unstale":"2024-01-21T03:56:19.706988+0000","last_undegraded":"2024-01-21T03:56:19.706988+0000","last_fullsized":"2024-01-21T03:56:19.706988+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:44:56.009576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1230,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.697115+0000","last_change":"2024-01-21T03:21:13.545831+0000","last_active":"2024-01-21T03:56:20.697115+0000","last_peered":"2024-01-21T03:56:20.697115+0000","last_clean":"2024-01-21T03:56:20.697115+0000","last_became_active":"2024-01-21T03:21:13.539461+0000","last_became_peered":"2024-01-21T03:21:13.539461+0000","last_unstale":"2024-01-21T03:56:20.697115+0000","last_undegraded":"2024-01-21T03:56:20.697115+0000","last_fullsized":"2024-01-21T03:56:20.697115+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:55:11.327991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1207,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.484590+0000","last_change":"2024-01-21T03:21:31.051184+0000","last_active":"2024-01-21T03:56:19.484590+0000","last_peered":"2024-01-21T03:56:19.484590+0000","last_clean":"2024-01-21T03:56:19.484590+0000","last_became_active":"2024-01-21T03:21:30.818768+0000","last_became_peered":"2024-01-21T03:21:30.818768+0000","last_unstale":"2024-01-21T03:56:19.484590+0000","last_undegraded":"2024-01-21T03:56:19.484590+0000","last_fullsized":"2024-01-21T03:56:19.484590+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:16:32.050608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1232,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.685544+0000","last_change":"2024-01-21T03:21:13.851032+0000","last_active":"2024-01-21T03:56:20.685544+0000","last_peered":"2024-01-21T03:56:20.685544+0000","last_clean":"2024-01-21T03:56:20.685544+0000","last_became_active":"2024-01-21T03:21:13.842398+0000","last_became_peered":"2024-01-21T03:21:13.842398+0000","last_unstale":"2024-01-21T03:56:20.685544+0000","last_undegraded":"2024-01-21T03:56:20.685544+0000","last_fullsized":"2024-01-21T03:56:20.685544+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:27:03.441063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1233,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.669592+0000","last_change":"2024-01-21T03:21:34.296238+0000","last_active":"2024-01-21T03:56:20.669592+0000","last_peered":"2024-01-21T03:56:20.669592+0000","last_clean":"2024-01-21T03:56:20.669592+0000","last_became_active":"2024-01-21T03:21:34.290896+0000","last_became_peered":"2024-01-21T03:21:34.290896+0000","last_unstale":"2024-01-21T03:56:20.669592+0000","last_undegraded":"2024-01-21T03:56:20.669592+0000","last_fullsized":"2024-01-21T03:56:20.669592+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:52:43.200440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1","version":"60'7","reported_seq":1224,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.681000+0000","last_change":"2024-01-21T03:21:35.777407+0000","last_active":"2024-01-21T03:56:20.681000+0000","last_peered":"2024-01-21T03:56:20.681000+0000","last_clean":"2024-01-21T03:56:20.681000+0000","last_became_active":"2024-01-21T03:21:35.763549+0000","last_became_peered":"2024-01-21T03:21:35.763549+0000","last_unstale":"2024-01-21T03:56:20.681000+0000","last_undegraded":"2024-01-21T03:56:20.681000+0000","last_fullsized":"2024-01-21T03:56:20.681000+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:56:17.661965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1023,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.957634+0000","last_change":"2024-01-21T03:26:06.764781+0000","last_active":"2024-01-21T03:56:24.957634+0000","last_peered":"2024-01-21T03:56:24.957634+0000","last_clean":"2024-01-21T03:56:24.957634+0000","last_became_active":"2024-01-21T03:26:06.758343+0000","last_became_peered":"2024-01-21T03:26:06.758343+0000","last_unstale":"2024-01-21T03:56:24.957634+0000","last_undegraded":"2024-01-21T03:56:24.957634+0000","last_fullsized":"2024-01-21T03:56:24.957634+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:09:42.970888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1226,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.259577+0000","last_change":"2024-01-21T03:21:15.810535+0000","last_active":"2024-01-21T03:56:20.259577+0000","last_peered":"2024-01-21T03:56:20.259577+0000","last_clean":"2024-01-21T03:56:20.259577+0000","last_became_active":"2024-01-21T03:21:15.779865+0000","last_became_peered":"2024-01-21T03:21:15.779865+0000","last_unstale":"2024-01-21T03:56:20.259577+0000","last_undegraded":"2024-01-21T03:56:20.259577+0000","last_fullsized":"2024-01-21T03:56:20.259577+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:56:54.491579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1207,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.502562+0000","last_change":"2024-01-21T03:21:22.661192+0000","last_active":"2024-01-21T03:56:19.502562+0000","last_peered":"2024-01-21T03:56:19.502562+0000","last_clean":"2024-01-21T03:56:19.502562+0000","last_became_active":"2024-01-21T03:21:22.647696+0000","last_became_peered":"2024-01-21T03:21:22.647696+0000","last_unstale":"2024-01-21T03:56:19.502562+0000","last_undegraded":"2024-01-21T03:56:19.502562+0000","last_fullsized":"2024-01-21T03:56:19.502562+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:05:43.481850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1218,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.275498+0000","last_change":"2024-01-21T03:21:26.231500+0000","last_active":"2024-01-21T03:56:20.275498+0000","last_peered":"2024-01-21T03:56:20.275498+0000","last_clean":"2024-01-21T03:56:20.275498+0000","last_became_active":"2024-01-21T03:21:26.223079+0000","last_became_peered":"2024-01-21T03:21:26.223079+0000","last_unstale":"2024-01-21T03:56:20.275498+0000","last_undegraded":"2024-01-21T03:56:20.275498+0000","last_fullsized":"2024-01-21T03:56:20.275498+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:36:33.965862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1232,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.261261+0000","last_change":"2024-01-21T03:26:06.699602+0000","last_active":"2024-01-21T03:56:20.261261+0000","last_peered":"2024-01-21T03:56:20.261261+0000","last_clean":"2024-01-21T03:56:20.261261+0000","last_became_active":"2024-01-21T03:26:06.693281+0000","last_became_peered":"2024-01-21T03:26:06.693281+0000","last_unstale":"2024-01-21T03:56:20.261261+0000","last_undegraded":"2024-01-21T03:56:20.261261+0000","last_fullsized":"2024-01-21T03:56:20.261261+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:29:07.025641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"23'76","reported_seq":1257,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.243138+0000","last_change":"2024-01-21T03:19:59.052204+0000","last_active":"2024-01-21T03:56:20.243138+0000","last_peered":"2024-01-21T03:56:20.243138+0000","last_clean":"2024-01-21T03:56:20.243138+0000","last_became_active":"2024-01-21T03:19:58.060095+0000","last_became_peered":"2024-01-21T03:19:58.060095+0000","last_unstale":"2024-01-21T03:56:20.243138+0000","last_undegraded":"2024-01-21T03:56:20.243138+0000","last_fullsized":"2024-01-21T03:56:20.243138+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:44.383554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:44.383554+0000","last_clean_scrub_stamp":"2024-01-21T03:19:44.383554+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:42:45.445502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1260,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.144254+0000","last_change":"2024-01-21T03:20:02.295222+0000","last_active":"2024-01-21T03:56:23.144254+0000","last_peered":"2024-01-21T03:56:23.144254+0000","last_clean":"2024-01-21T03:56:23.144254+0000","last_became_active":"2024-01-21T03:20:02.289915+0000","last_became_peered":"2024-01-21T03:20:02.289915+0000","last_unstale":"2024-01-21T03:56:23.144254+0000","last_undegraded":"2024-01-21T03:56:23.144254+0000","last_fullsized":"2024-01-21T03:56:23.144254+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:36:11.210361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1218,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.272174+0000","last_change":"2024-01-21T03:21:14.853643+0000","last_active":"2024-01-21T03:56:20.272174+0000","last_peered":"2024-01-21T03:56:20.272174+0000","last_clean":"2024-01-21T03:56:20.272174+0000","last_became_active":"2024-01-21T03:21:14.829810+0000","last_became_peered":"2024-01-21T03:21:14.829810+0000","last_unstale":"2024-01-21T03:56:20.272174+0000","last_undegraded":"2024-01-21T03:56:20.272174+0000","last_fullsized":"2024-01-21T03:56:20.272174+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:30:12.597754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1261,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.105259+0000","last_change":"2024-01-21T03:20:01.812917+0000","last_active":"2024-01-21T03:56:23.105259+0000","last_peered":"2024-01-21T03:56:23.105259+0000","last_clean":"2024-01-21T03:56:23.105259+0000","last_became_active":"2024-01-21T03:20:01.784281+0000","last_became_peered":"2024-01-21T03:20:01.784281+0000","last_unstale":"2024-01-21T03:56:23.105259+0000","last_undegraded":"2024-01-21T03:56:23.105259+0000","last_fullsized":"2024-01-21T03:56:23.105259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:26:56.084862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1254,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.175368+0000","last_change":"2024-01-21T03:21:26.074817+0000","last_active":"2024-01-21T03:56:23.175368+0000","last_peered":"2024-01-21T03:56:23.175368+0000","last_clean":"2024-01-21T03:56:23.175368+0000","last_became_active":"2024-01-21T03:21:26.057996+0000","last_became_peered":"2024-01-21T03:21:26.057996+0000","last_unstale":"2024-01-21T03:56:23.175368+0000","last_undegraded":"2024-01-21T03:56:23.175368+0000","last_fullsized":"2024-01-21T03:56:23.175368+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:22:40.041984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1216,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.674340+0000","last_change":"2024-01-21T03:20:01.846748+0000","last_active":"2024-01-21T03:56:20.674340+0000","last_peered":"2024-01-21T03:56:20.674340+0000","last_clean":"2024-01-21T03:56:20.674340+0000","last_became_active":"2024-01-21T03:20:01.818592+0000","last_became_peered":"2024-01-21T03:20:01.818592+0000","last_unstale":"2024-01-21T03:56:20.674340+0000","last_undegraded":"2024-01-21T03:56:20.674340+0000","last_fullsized":"2024-01-21T03:56:20.674340+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:08:09.861694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1226,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.257372+0000","last_change":"2024-01-21T03:22:06.056192+0000","last_active":"2024-01-21T03:56:20.257372+0000","last_peered":"2024-01-21T03:56:20.257372+0000","last_clean":"2024-01-21T03:56:20.257372+0000","last_became_active":"2024-01-21T03:22:06.040269+0000","last_became_peered":"2024-01-21T03:22:06.040269+0000","last_unstale":"2024-01-21T03:56:20.257372+0000","last_undegraded":"2024-01-21T03:56:20.257372+0000","last_fullsized":"2024-01-21T03:56:20.257372+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:36:24.196035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1260,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.122291+0000","last_change":"2024-01-21T03:20:02.210632+0000","last_active":"2024-01-21T03:56:23.122291+0000","last_peered":"2024-01-21T03:56:23.122291+0000","last_clean":"2024-01-21T03:56:23.122291+0000","last_became_active":"2024-01-21T03:20:02.201314+0000","last_became_peered":"2024-01-21T03:20:02.201314+0000","last_unstale":"2024-01-21T03:56:23.122291+0000","last_undegraded":"2024-01-21T03:56:23.122291+0000","last_fullsized":"2024-01-21T03:56:23.122291+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:55:53.424403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1232,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.668006+0000","last_change":"2024-01-21T03:21:34.314505+0000","last_active":"2024-01-21T03:56:20.668006+0000","last_peered":"2024-01-21T03:56:20.668006+0000","last_clean":"2024-01-21T03:56:20.668006+0000","last_became_active":"2024-01-21T03:21:34.307508+0000","last_became_peered":"2024-01-21T03:21:34.307508+0000","last_unstale":"2024-01-21T03:56:20.668006+0000","last_undegraded":"2024-01-21T03:56:20.668006+0000","last_fullsized":"2024-01-21T03:56:20.668006+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:38:13.727492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1224,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.273664+0000","last_change":"2024-01-21T03:20:02.214415+0000","last_active":"2024-01-21T03:56:20.273664+0000","last_peered":"2024-01-21T03:56:20.273664+0000","last_clean":"2024-01-21T03:56:20.273664+0000","last_became_active":"2024-01-21T03:20:02.207814+0000","last_became_peered":"2024-01-21T03:20:02.207814+0000","last_unstale":"2024-01-21T03:56:20.273664+0000","last_undegraded":"2024-01-21T03:56:20.273664+0000","last_fullsized":"2024-01-21T03:56:20.273664+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:12:28.359349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1215,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.955788+0000","last_change":"2024-01-21T03:21:23.773815+0000","last_active":"2024-01-21T03:56:24.955788+0000","last_peered":"2024-01-21T03:56:24.955788+0000","last_clean":"2024-01-21T03:56:24.955788+0000","last_became_active":"2024-01-21T03:21:23.767027+0000","last_became_peered":"2024-01-21T03:21:23.767027+0000","last_unstale":"2024-01-21T03:56:24.955788+0000","last_undegraded":"2024-01-21T03:56:24.955788+0000","last_fullsized":"2024-01-21T03:56:24.955788+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:56:41.913140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1260,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.171699+0000","last_change":"2024-01-21T03:20:01.878033+0000","last_active":"2024-01-21T03:56:23.171699+0000","last_peered":"2024-01-21T03:56:23.171699+0000","last_clean":"2024-01-21T03:56:23.171699+0000","last_became_active":"2024-01-21T03:20:01.872724+0000","last_became_peered":"2024-01-21T03:20:01.872724+0000","last_unstale":"2024-01-21T03:56:23.171699+0000","last_undegraded":"2024-01-21T03:56:23.171699+0000","last_fullsized":"2024-01-21T03:56:23.171699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:06:41.515884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1207,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.486663+0000","last_change":"2024-01-21T03:21:55.470689+0000","last_active":"2024-01-21T03:56:19.486663+0000","last_peered":"2024-01-21T03:56:19.486663+0000","last_clean":"2024-01-21T03:56:19.486663+0000","last_became_active":"2024-01-21T03:21:55.460120+0000","last_became_peered":"2024-01-21T03:21:55.460120+0000","last_unstale":"2024-01-21T03:56:19.486663+0000","last_undegraded":"2024-01-21T03:56:19.486663+0000","last_fullsized":"2024-01-21T03:56:19.486663+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:03:53.385543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1236,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.701374+0000","last_change":"2024-01-21T03:20:02.260437+0000","last_active":"2024-01-21T03:56:20.701374+0000","last_peered":"2024-01-21T03:56:20.701374+0000","last_clean":"2024-01-21T03:56:20.701374+0000","last_became_active":"2024-01-21T03:20:02.234092+0000","last_became_peered":"2024-01-21T03:20:02.234092+0000","last_unstale":"2024-01-21T03:56:20.701374+0000","last_undegraded":"2024-01-21T03:56:20.701374+0000","last_fullsized":"2024-01-21T03:56:20.701374+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:32:42.605439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1232,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.703783+0000","last_change":"2024-01-21T03:21:09.536570+0000","last_active":"2024-01-21T03:56:19.703783+0000","last_peered":"2024-01-21T03:56:19.703783+0000","last_clean":"2024-01-21T03:56:19.703783+0000","last_became_active":"2024-01-21T03:21:09.531293+0000","last_became_peered":"2024-01-21T03:21:09.531293+0000","last_unstale":"2024-01-21T03:56:19.703783+0000","last_undegraded":"2024-01-21T03:56:19.703783+0000","last_fullsized":"2024-01-21T03:56:19.703783+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:01:17.017524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1256,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.197738+0000","last_change":"2024-01-21T03:20:02.233166+0000","last_active":"2024-01-21T03:56:23.197738+0000","last_peered":"2024-01-21T03:56:23.197738+0000","last_clean":"2024-01-21T03:56:23.197738+0000","last_became_active":"2024-01-21T03:20:02.226975+0000","last_became_peered":"2024-01-21T03:20:02.226975+0000","last_unstale":"2024-01-21T03:56:23.197738+0000","last_undegraded":"2024-01-21T03:56:23.197738+0000","last_fullsized":"2024-01-21T03:56:23.197738+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:36:45.013749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1253,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.119814+0000","last_change":"2024-01-21T03:21:36.938379+0000","last_active":"2024-01-21T03:56:23.119814+0000","last_peered":"2024-01-21T03:56:23.119814+0000","last_clean":"2024-01-21T03:56:23.119814+0000","last_became_active":"2024-01-21T03:21:36.929227+0000","last_became_peered":"2024-01-21T03:21:36.929227+0000","last_unstale":"2024-01-21T03:56:23.119814+0000","last_undegraded":"2024-01-21T03:56:23.119814+0000","last_fullsized":"2024-01-21T03:56:23.119814+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:55:39.626168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1218,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.252018+0000","last_change":"2024-01-21T03:21:13.975490+0000","last_active":"2024-01-21T03:56:20.252018+0000","last_peered":"2024-01-21T03:56:20.252018+0000","last_clean":"2024-01-21T03:56:20.252018+0000","last_became_active":"2024-01-21T03:21:13.969066+0000","last_became_peered":"2024-01-21T03:21:13.969066+0000","last_unstale":"2024-01-21T03:56:20.252018+0000","last_undegraded":"2024-01-21T03:56:20.252018+0000","last_fullsized":"2024-01-21T03:56:20.252018+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:53:18.899854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1231,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.705306+0000","last_change":"2024-01-21T03:21:22.338167+0000","last_active":"2024-01-21T03:56:19.705306+0000","last_peered":"2024-01-21T03:56:19.705306+0000","last_clean":"2024-01-21T03:56:19.705306+0000","last_became_active":"2024-01-21T03:21:22.319120+0000","last_became_peered":"2024-01-21T03:21:22.319120+0000","last_unstale":"2024-01-21T03:56:19.705306+0000","last_undegraded":"2024-01-21T03:56:19.705306+0000","last_fullsized":"2024-01-21T03:56:19.705306+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:35:43.974713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1230,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.681744+0000","last_change":"2024-01-21T03:21:17.587236+0000","last_active":"2024-01-21T03:56:20.681744+0000","last_peered":"2024-01-21T03:56:20.681744+0000","last_clean":"2024-01-21T03:56:20.681744+0000","last_became_active":"2024-01-21T03:21:17.582134+0000","last_became_peered":"2024-01-21T03:21:17.582134+0000","last_unstale":"2024-01-21T03:56:20.681744+0000","last_undegraded":"2024-01-21T03:56:20.681744+0000","last_fullsized":"2024-01-21T03:56:20.681744+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:14:12.050155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1253,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.167852+0000","last_change":"2024-01-21T03:21:40.581757+0000","last_active":"2024-01-21T03:56:23.167852+0000","last_peered":"2024-01-21T03:56:23.167852+0000","last_clean":"2024-01-21T03:56:23.167852+0000","last_became_active":"2024-01-21T03:21:40.568805+0000","last_became_peered":"2024-01-21T03:21:40.568805+0000","last_unstale":"2024-01-21T03:56:23.167852+0000","last_undegraded":"2024-01-21T03:56:23.167852+0000","last_fullsized":"2024-01-21T03:56:23.167852+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:33:02.221661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1226,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.588370+0000","last_change":"2024-01-21T03:22:06.157185+0000","last_active":"2024-01-21T03:56:20.588370+0000","last_peered":"2024-01-21T03:56:20.588370+0000","last_clean":"2024-01-21T03:56:20.588370+0000","last_became_active":"2024-01-21T03:22:06.149457+0000","last_became_peered":"2024-01-21T03:22:06.149457+0000","last_unstale":"2024-01-21T03:56:20.588370+0000","last_undegraded":"2024-01-21T03:56:20.588370+0000","last_fullsized":"2024-01-21T03:56:20.588370+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:47:38.535776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1230,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.721033+0000","last_change":"2024-01-21T03:21:13.379783+0000","last_active":"2024-01-21T03:56:20.721033+0000","last_peered":"2024-01-21T03:56:20.721033+0000","last_clean":"2024-01-21T03:56:20.721033+0000","last_became_active":"2024-01-21T03:21:13.373828+0000","last_became_peered":"2024-01-21T03:21:13.373828+0000","last_unstale":"2024-01-21T03:56:20.721033+0000","last_undegraded":"2024-01-21T03:56:20.721033+0000","last_fullsized":"2024-01-21T03:56:20.721033+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:54:16.898824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1229,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.720120+0000","last_change":"2024-01-21T03:21:11.007665+0000","last_active":"2024-01-21T03:56:19.720120+0000","last_peered":"2024-01-21T03:56:19.720120+0000","last_clean":"2024-01-21T03:56:19.720120+0000","last_became_active":"2024-01-21T03:21:10.999264+0000","last_became_peered":"2024-01-21T03:21:10.999264+0000","last_unstale":"2024-01-21T03:56:19.720120+0000","last_undegraded":"2024-01-21T03:56:19.720120+0000","last_fullsized":"2024-01-21T03:56:19.720120+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:43:27.507683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1226,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.245990+0000","last_change":"2024-01-21T03:21:15.387706+0000","last_active":"2024-01-21T03:56:20.245990+0000","last_peered":"2024-01-21T03:56:20.245990+0000","last_clean":"2024-01-21T03:56:20.245990+0000","last_became_active":"2024-01-21T03:21:15.068960+0000","last_became_peered":"2024-01-21T03:21:15.068960+0000","last_unstale":"2024-01-21T03:56:20.245990+0000","last_undegraded":"2024-01-21T03:56:20.245990+0000","last_fullsized":"2024-01-21T03:56:20.245990+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:01:23.405734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1232,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.696894+0000","last_change":"2024-01-21T03:21:33.019724+0000","last_active":"2024-01-21T03:56:19.696894+0000","last_peered":"2024-01-21T03:56:19.696894+0000","last_clean":"2024-01-21T03:56:19.696894+0000","last_became_active":"2024-01-21T03:21:32.997061+0000","last_became_peered":"2024-01-21T03:21:32.997061+0000","last_unstale":"2024-01-21T03:56:19.696894+0000","last_undegraded":"2024-01-21T03:56:19.696894+0000","last_fullsized":"2024-01-21T03:56:19.696894+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:52:53.090792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1218,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.266819+0000","last_change":"2024-01-21T03:21:14.774673+0000","last_active":"2024-01-21T03:56:20.266819+0000","last_peered":"2024-01-21T03:56:20.266819+0000","last_clean":"2024-01-21T03:56:20.266819+0000","last_became_active":"2024-01-21T03:21:14.761665+0000","last_became_peered":"2024-01-21T03:21:14.761665+0000","last_unstale":"2024-01-21T03:56:20.266819+0000","last_undegraded":"2024-01-21T03:56:20.266819+0000","last_fullsized":"2024-01-21T03:56:20.266819+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:09:47.334671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1232,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.715724+0000","last_change":"2024-01-21T03:21:34.181963+0000","last_active":"2024-01-21T03:56:19.715724+0000","last_peered":"2024-01-21T03:56:19.715724+0000","last_clean":"2024-01-21T03:56:19.715724+0000","last_became_active":"2024-01-21T03:21:34.168272+0000","last_became_peered":"2024-01-21T03:21:34.168272+0000","last_unstale":"2024-01-21T03:56:19.715724+0000","last_undegraded":"2024-01-21T03:56:19.715724+0000","last_fullsized":"2024-01-21T03:56:19.715724+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:16:18.395527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280948,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76408,"kb_used_data":4964,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295432,"statfs":{"total":96636764160,"available":96558522368,"internally_reserved":0,"allocated":5083136,"data_stored":2325045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":9.3070000000000004,"5min":22.673999999999999,"15min":23.361000000000001},"min":{"1min":5.9930000000000003,"5min":4.7569999999999997,"15min":4.2290000000000001},"max":{"1min":15.645,"5min":221.726,"15min":236.40199999999999},"last":8.2629999999999999},{"interface":"front","average":{"1min":12.923,"5min":28.297999999999998,"15min":22.507000000000001},"min":{"1min":7.3280000000000003,"5min":6.5869999999999997,"15min":4.423},"max":{"1min":21.279,"5min":315.84899999999999,"15min":317.16500000000002},"last":11.234}]},{"osd":1,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":12.68,"5min":24.010999999999999,"15min":26.009},"min":{"1min":7.8280000000000003,"5min":7.8280000000000003,"15min":6.7649999999999997},"max":{"1min":19.681999999999999,"5min":210.08799999999999,"15min":688.34500000000003},"last":11.881},{"interface":"front","average":{"1min":15.066000000000001,"5min":30.968,"15min":31.401},"min":{"1min":10.843999999999999,"5min":10.49,"15min":9.6310000000000002},"max":{"1min":23.169,"5min":288.46600000000001,"15min":686.94299999999998},"last":17.687999999999999}]},{"osd":2,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":9.9570000000000007,"5min":59.222999999999999,"15min":34.966999999999999},"min":{"1min":6.0730000000000004,"5min":6.0730000000000004,"15min":5.75},"max":{"1min":31.401,"5min":3500.2379999999998,"15min":3500.2379999999998},"last":8.6159999999999997},{"interface":"front","average":{"1min":14.031000000000001,"5min":37.201000000000001,"15min":29.867000000000001},"min":{"1min":7.9669999999999996,"5min":7.9669999999999996,"15min":7.4710000000000001},"max":{"1min":25.942,"5min":919.58199999999999,"15min":919.58199999999999},"last":13.473000000000001}]},{"osd":4,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":14.010999999999999,"5min":26.649000000000001,"15min":26.817},"min":{"1min":10.026,"5min":9.1489999999999991,"15min":8.9909999999999997},"max":{"1min":18.762,"5min":241.95599999999999,"15min":297.46600000000001},"last":12.843999999999999},{"interface":"front","average":{"1min":18.001999999999999,"5min":31.356999999999999,"15min":31.646999999999998},"min":{"1min":12.919,"5min":12.346,"15min":9.673},"max":{"1min":23.687000000000001,"5min":292.25400000000002,"15min":340.73500000000001},"last":19.527999999999999}]},{"osd":5,"last update":"Sun Jan 21 03:56:18 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":24.042999999999999,"15min":24.727},"min":{"1min":7.3680000000000003,"5min":7.3680000000000003,"15min":6.1989999999999998},"max":{"1min":22.300000000000001,"5min":214.58500000000001,"15min":362.22899999999998},"last":14.112},{"interface":"front","average":{"1min":13.831,"5min":26.408000000000001,"15min":27.535},"min":{"1min":8.9079999999999995,"5min":6.5439999999999996,"15min":6.5439999999999996},"max":{"1min":21.576000000000001,"5min":339.19499999999999,"15min":339.19499999999999},"last":16.254999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":18.814,"5min":31.257000000000001,"15min":30.303999999999998},"min":{"1min":11.645,"5min":11.206,"15min":7.1719999999999997},"max":{"1min":42.293999999999997,"5min":211.822,"15min":572.20899999999995},"last":16.989000000000001},{"interface":"front","average":{"1min":21.713000000000001,"5min":36.789999999999999,"15min":34.954999999999998},"min":{"1min":13.535,"5min":13.535,"15min":9.7240000000000002},"max":{"1min":45.457999999999998,"5min":215.315,"15min":376.66399999999999},"last":20.728000000000002}]},{"osd":7,"last update":"Sun Jan 21 03:56:18 2024","interfaces":[{"interface":"back","average":{"1min":15.071999999999999,"5min":23.681999999999999,"15min":23.119},"min":{"1min":9.3960000000000008,"5min":8.1329999999999991,"15min":5.9169999999999998},"max":{"1min":30.167999999999999,"5min":213.73500000000001,"15min":274.91000000000003},"last":10.551},{"interface":"front","average":{"1min":21.024000000000001,"5min":32.463999999999999,"15min":32.999000000000002},"min":{"1min":14.153,"5min":11.337,"15min":11.337},"max":{"1min":31.466000000000001,"5min":234.46299999999999,"15min":339.613},"last":21.956}]}]},{"osd":2,"up_from":21,"seq":90194313656,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68040,"kb_used_data":4924,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303800,"statfs":{"total":96636764160,"available":96567091200,"internally_reserved":0,"allocated":5042176,"data_stored":2303320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:55:51 2024","interfaces":[{"interface":"back","average":{"1min":11.087,"5min":20.286999999999999,"15min":17.802},"min":{"1min":4.9820000000000002,"5min":4.9820000000000002,"15min":4.5460000000000003},"max":{"1min":23.617000000000001,"5min":335.38200000000001,"15min":335.38200000000001},"last":5.5350000000000001},{"interface":"front","average":{"1min":9.5939999999999994,"5min":20.925999999999998,"15min":17.704000000000001},"min":{"1min":5.484,"5min":5.484,"15min":4.4580000000000002},"max":{"1min":22.068999999999999,"5min":343.05599999999998,"15min":343.05599999999998},"last":7.6950000000000003}]},{"osd":1,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":16.661999999999999,"5min":32.128,"15min":25.600999999999999},"min":{"1min":7.641,"5min":5.5960000000000001,"15min":5.2190000000000003},"max":{"1min":50.104999999999997,"5min":341.34399999999999,"15min":504.58100000000002},"last":9.5600000000000005},{"interface":"front","average":{"1min":14.928000000000001,"5min":35.445,"15min":27.994},"min":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":5.7350000000000003},"max":{"1min":32.649000000000001,"5min":674.13400000000001,"15min":674.13400000000001},"last":16.646000000000001}]},{"osd":3,"last update":"Sun Jan 21 03:55:37 2024","interfaces":[{"interface":"back","average":{"1min":14.738,"5min":22.449999999999999,"15min":20.140999999999998},"min":{"1min":7.9509999999999996,"5min":6.9820000000000002,"15min":6.3659999999999997},"max":{"1min":45.518999999999998,"5min":263.31400000000002,"15min":263.31400000000002},"last":12.964},{"interface":"front","average":{"1min":17.158999999999999,"5min":26.888000000000002,"15min":22.359999999999999},"min":{"1min":9.6389999999999993,"5min":8.1180000000000003,"15min":8.1180000000000003},"max":{"1min":50.167000000000002,"5min":683.75800000000004,"15min":683.75800000000004},"last":17.747}]},{"osd":4,"last update":"Sun Jan 21 03:55:26 2024","interfaces":[{"interface":"back","average":{"1min":17.934999999999999,"5min":27.861000000000001,"15min":24.173999999999999},"min":{"1min":9.5340000000000007,"5min":8.827,"15min":8.0609999999999999},"max":{"1min":34.628,"5min":687.15800000000002,"15min":687.15800000000002},"last":10.741},{"interface":"front","average":{"1min":19.096,"5min":24.669,"15min":22.831},"min":{"1min":11.587,"5min":8.8379999999999992,"15min":8.1620000000000008},"max":{"1min":41.676000000000002,"5min":318.77199999999999,"15min":318.77199999999999},"last":18.603000000000002}]},{"osd":5,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":19.896000000000001,"5min":30.462,"15min":24.710999999999999},"min":{"1min":7.9740000000000002,"5min":7.6660000000000004,"15min":6.0650000000000004},"max":{"1min":50.735999999999997,"5min":348.47000000000003,"15min":348.47000000000003},"last":14.537000000000001},{"interface":"front","average":{"1min":21.834,"5min":30.434999999999999,"15min":25.710000000000001},"min":{"1min":10.997999999999999,"5min":10.967000000000001,"15min":8.5210000000000008},"max":{"1min":52.564999999999998,"5min":264.41199999999998,"15min":264.41199999999998},"last":11.855}]},{"osd":6,"last update":"Sun Jan 21 03:55:26 2024","interfaces":[{"interface":"back","average":{"1min":19.821000000000002,"5min":31.231999999999999,"15min":27.202000000000002},"min":{"1min":12.441000000000001,"5min":8.5960000000000001,"15min":7.2800000000000002},"max":{"1min":52.945,"5min":679.51900000000001,"15min":679.51900000000001},"last":19.686},{"interface":"front","average":{"1min":21.942,"5min":34.5,"15min":28.763999999999999},"min":{"1min":14.582000000000001,"5min":10.416,"15min":10.121},"max":{"1min":53.688000000000002,"5min":676.11800000000005,"15min":676.11800000000005},"last":20.606999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":24.338999999999999,"5min":29.204999999999998,"15min":25.914000000000001},"min":{"1min":12.472,"5min":10.278,"15min":10.278},"max":{"1min":53.186999999999998,"5min":261.48899999999998,"15min":261.48899999999998},"last":13.686999999999999},{"interface":"front","average":{"1min":24.440000000000001,"5min":38.350999999999999,"15min":27.709},"min":{"1min":15.018000000000001,"5min":12.605,"15min":8.4239999999999995},"max":{"1min":42.475000000000001,"5min":678.09500000000003,"15min":678.09500000000003},"last":21.939}]}]},{"osd":1,"up_from":21,"seq":90194313655,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94744,"kb_used_data":5316,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94277096,"statfs":{"total":96636764160,"available":96539746304,"internally_reserved":0,"allocated":5443584,"data_stored":2701160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":11.568,"5min":16.558,"15min":21.972999999999999},"min":{"1min":4.5970000000000004,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":26.126000000000001,"5min":154.52799999999999,"15min":528.255},"last":6.4720000000000004},{"interface":"front","average":{"1min":12.198,"5min":15.057,"15min":19.050999999999998},"min":{"1min":4.9619999999999997,"5min":4.5199999999999996,"15min":4.3079999999999998},"max":{"1min":24.762,"5min":127.73999999999999,"15min":447.83199999999999},"last":5.5960000000000001}]},{"osd":2,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":13.651,"5min":17.736999999999998,"15min":23.350999999999999},"min":{"1min":8.5549999999999997,"5min":7.8719999999999999,"15min":6.194},"max":{"1min":21.834,"5min":116.48999999999999,"15min":420.87700000000001},"last":11.682},{"interface":"front","average":{"1min":15.859999999999999,"5min":26.047999999999998,"15min":27.696999999999999},"min":{"1min":8.5239999999999991,"5min":7.9889999999999999,"15min":7.3209999999999997},"max":{"1min":30.548999999999999,"5min":265.24200000000002,"15min":451.637},"last":12.586}]},{"osd":3,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":20.800000000000001,"5min":22.111000000000001,"15min":23.620999999999999},"min":{"1min":9.2140000000000004,"5min":8.0370000000000008,"15min":8.0370000000000008},"max":{"1min":62.801000000000002,"5min":141.52099999999999,"15min":419.892},"last":13.31},{"interface":"front","average":{"1min":28.942,"5min":25.436,"15min":27.707000000000001},"min":{"1min":8.9109999999999996,"5min":8.6479999999999997,"15min":8.3059999999999992},"max":{"1min":123.798,"5min":143.58699999999999,"15min":425.76100000000002},"last":18.611999999999998}]},{"osd":4,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":26.920999999999999,"5min":26.507999999999999,"15min":25.122},"min":{"1min":11.009,"5min":7.4210000000000003,"15min":7.4210000000000003},"max":{"1min":106.116,"5min":229.63200000000001,"15min":344.48200000000003},"last":13.99},{"interface":"front","average":{"1min":25.748999999999999,"5min":29.904,"15min":30.959},"min":{"1min":12.282,"5min":8.3059999999999992,"15min":8.1259999999999994},"max":{"1min":65.070999999999998,"5min":236.571,"15min":385.43000000000001},"last":19.704999999999998}]},{"osd":5,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":22.766999999999999,"5min":28.076000000000001,"15min":30.946000000000002},"min":{"1min":13.015000000000001,"5min":9.5839999999999996,"15min":9.5839999999999996},"max":{"1min":38.545999999999999,"5min":138.84899999999999,"15min":338.88200000000001},"last":20.771000000000001},{"interface":"front","average":{"1min":18.858000000000001,"5min":26.266999999999999,"15min":29.963999999999999},"min":{"1min":7.577,"5min":7.577,"15min":7.577},"max":{"1min":34.542999999999999,"5min":187.03,"15min":356.423},"last":14.676}]},{"osd":6,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":21.745000000000001,"5min":31.524000000000001,"15min":31.725000000000001},"min":{"1min":10.369999999999999,"5min":10.369999999999999,"15min":9.7289999999999992},"max":{"1min":53.404000000000003,"5min":280.85000000000002,"15min":409.08100000000002},"last":15.343999999999999},{"interface":"front","average":{"1min":20.478000000000002,"5min":25.875,"15min":23.920000000000002},"min":{"1min":6.6100000000000003,"5min":6.6100000000000003,"15min":6.6100000000000003},"max":{"1min":59.128999999999998,"5min":243.083,"15min":261.75099999999998},"last":9.7629999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":21.91,"5min":28.564,"15min":25.913},"min":{"1min":12.441000000000001,"5min":11.923,"15min":7.21},"max":{"1min":43.122999999999998,"5min":169.571,"15min":357.57299999999998},"last":16.021000000000001},{"interface":"front","average":{"1min":23.428000000000001,"5min":23.902999999999999,"15min":25.715},"min":{"1min":8.6929999999999996,"5min":8.1959999999999997,"15min":7.9729999999999999},"max":{"1min":63.999000000000002,"5min":176.809,"15min":418.70600000000002},"last":10.644}]}]},{"osd":0,"up_from":21,"seq":90194313659,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72244,"kb_used_data":4964,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299596,"statfs":{"total":96636764160,"available":96562786304,"internally_reserved":0,"allocated":5083136,"data_stored":2325045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":15.691000000000001,"5min":26.472999999999999,"15min":22.228000000000002},"min":{"1min":6.7560000000000002,"5min":6.0220000000000002,"15min":5.7869999999999999},"max":{"1min":49.520000000000003,"5min":246.35400000000001,"15min":301.80599999999998},"last":8.3079999999999998},{"interface":"front","average":{"1min":21.231000000000002,"5min":29.074000000000002,"15min":24.678000000000001},"min":{"1min":11.365,"5min":6.0270000000000001,"15min":6.0270000000000001},"max":{"1min":51.290999999999997,"5min":232.851,"15min":242.80000000000001},"last":21.334}]},{"osd":2,"last update":"Sun Jan 21 03:55:44 2024","interfaces":[{"interface":"back","average":{"1min":16.657,"5min":32.222999999999999,"15min":27.050999999999998},"min":{"1min":9.2360000000000007,"5min":7.2160000000000002,"15min":6.6289999999999996},"max":{"1min":28.940000000000001,"5min":279.71300000000002,"15min":312.15800000000002},"last":18.870000000000001},{"interface":"front","average":{"1min":12.459,"5min":35.030999999999999,"15min":27.09},"min":{"1min":6.9169999999999998,"5min":6.3959999999999999,"15min":5.0910000000000002},"max":{"1min":23.015000000000001,"5min":332.57499999999999,"15min":357.38900000000001},"last":17.791}]},{"osd":3,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":22.167999999999999,"5min":43.600000000000001,"15min":32.090000000000003},"min":{"1min":7.5049999999999999,"5min":7.5049999999999999,"15min":6.9829999999999997},"max":{"1min":68.963999999999999,"5min":547.15800000000002,"15min":547.15800000000002},"last":20.677},{"interface":"front","average":{"1min":20.045999999999999,"5min":44.688000000000002,"15min":32.734999999999999},"min":{"1min":10.461,"5min":7.5979999999999999,"15min":7.5979999999999999},"max":{"1min":74.480999999999995,"5min":361.11500000000001,"15min":361.11500000000001},"last":26.079999999999998}]},{"osd":4,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":22.556000000000001,"5min":35.131,"15min":25.462},"min":{"1min":10.24,"5min":7.4630000000000001,"15min":7.3019999999999996},"max":{"1min":75.359999999999999,"5min":333.23899999999998,"15min":333.23899999999998},"last":27.132999999999999},{"interface":"front","average":{"1min":27.826000000000001,"5min":41.021000000000001,"15min":28.832999999999998},"min":{"1min":14.176,"5min":10.836,"15min":6.3630000000000004},"max":{"1min":90.658000000000001,"5min":333.87400000000002,"15min":333.87400000000002},"last":29.652000000000001}]},{"osd":5,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":12.843,"5min":40.453000000000003,"15min":31.494},"min":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"max":{"1min":66.775999999999996,"5min":387.58100000000002,"15min":543.84100000000001},"last":6.617},{"interface":"front","average":{"1min":26.053999999999998,"5min":30.795999999999999,"15min":23.568000000000001},"min":{"1min":10.856999999999999,"5min":6.3529999999999998,"15min":5.452},"max":{"1min":88.313999999999993,"5min":266.03800000000001,"15min":335.589},"last":23.265000000000001}]},{"osd":6,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":16.995999999999999,"5min":31.850000000000001,"15min":26.893999999999998},"min":{"1min":9.0709999999999997,"5min":6.1059999999999999,"15min":6.1059999999999999},"max":{"1min":67.790999999999997,"5min":311.358,"15min":370.30200000000002},"last":30.492000000000001},{"interface":"front","average":{"1min":25.103000000000002,"5min":33.008000000000003,"15min":26.288},"min":{"1min":12.054,"5min":8.2769999999999992,"15min":7.2389999999999999},"max":{"1min":70.632000000000005,"5min":230.27699999999999,"15min":230.27699999999999},"last":19.509}]},{"osd":7,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":24.888000000000002,"5min":30.061,"15min":25.626000000000001},"min":{"1min":10.714,"5min":9.1110000000000007,"15min":8.8960000000000008},"max":{"1min":71.314999999999998,"5min":131.65000000000001,"15min":214.071},"last":16.683},{"interface":"front","average":{"1min":28.067,"5min":38.735999999999997,"15min":28.626000000000001},"min":{"1min":12.391,"5min":8.8070000000000004,"15min":8.8070000000000004},"max":{"1min":94.670000000000002,"5min":348.38600000000002,"15min":348.38600000000002},"last":25.440999999999999}]}]},{"osd":6,"up_from":18,"seq":77309411770,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72208,"kb_used_data":4928,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299632,"statfs":{"total":96636764160,"available":96562823168,"internally_reserved":0,"allocated":5046272,"data_stored":2303325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":16.803000000000001,"5min":22.076000000000001,"15min":21.289999999999999},"min":{"1min":5.3380000000000001,"5min":5.3380000000000001,"15min":4.391},"max":{"1min":98.180000000000007,"5min":284.42500000000001,"15min":438.98899999999998},"last":12.811},{"interface":"front","average":{"1min":28.359999999999999,"5min":27.478999999999999,"15min":21.431000000000001},"min":{"1min":8.7149999999999999,"5min":5.2160000000000002,"15min":4.8840000000000003},"max":{"1min":144.92400000000001,"5min":380.07600000000002,"15min":380.07600000000002},"last":10.606999999999999}]},{"osd":1,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":19.898,"5min":31.847999999999999,"15min":24.283000000000001},"min":{"1min":8.7289999999999992,"5min":6.883,"15min":4.7389999999999999},"max":{"1min":54.722999999999999,"5min":526.48099999999999,"15min":526.48099999999999},"last":7.7670000000000003},{"interface":"front","average":{"1min":14.859999999999999,"5min":26.492000000000001,"15min":22.867999999999999},"min":{"1min":6.3600000000000003,"5min":4.3120000000000003,"15min":4.3120000000000003},"max":{"1min":44.970999999999997,"5min":395.72399999999999,"15min":395.72399999999999},"last":9.3170000000000002}]},{"osd":2,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":12.648,"5min":21.052,"15min":20.300999999999998},"min":{"1min":6.0350000000000001,"5min":5.8920000000000003,"15min":5.8920000000000003},"max":{"1min":35.268999999999998,"5min":209.78,"15min":213.83199999999999},"last":12.169},{"interface":"front","average":{"1min":22.007000000000001,"5min":31.407,"15min":26.654},"min":{"1min":6.7210000000000001,"5min":6.7210000000000001,"15min":6.7210000000000001},"max":{"1min":77.159000000000006,"5min":444.822,"15min":444.822},"last":26.718}]},{"osd":3,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":18.542999999999999,"5min":33.561,"15min":26.021000000000001},"min":{"1min":8.1959999999999997,"5min":6.1890000000000001,"15min":6.1890000000000001},"max":{"1min":59.645000000000003,"5min":527.40899999999999,"15min":527.40899999999999},"last":27.533000000000001},{"interface":"front","average":{"1min":17.805,"5min":25.48,"15min":23.213999999999999},"min":{"1min":6.4089999999999998,"5min":6.4050000000000002,"15min":6.4050000000000002},"max":{"1min":57.729999999999997,"5min":370.10000000000002,"15min":370.10000000000002},"last":14.664999999999999}]},{"osd":4,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":21.780000000000001,"5min":31.358000000000001,"15min":24.521999999999998},"min":{"1min":6.9160000000000004,"5min":6.3120000000000003,"15min":6.3120000000000003},"max":{"1min":95.962000000000003,"5min":385.49200000000002,"15min":385.49200000000002},"last":6.5410000000000004},{"interface":"front","average":{"1min":21.581,"5min":26.248000000000001,"15min":21.954000000000001},"min":{"1min":8.0660000000000007,"5min":7.2130000000000001,"15min":6.7539999999999996},"max":{"1min":89.241,"5min":241.59200000000001,"15min":241.59200000000001},"last":9.9629999999999992}]},{"osd":5,"last update":"Sun Jan 21 03:55:30 2024","interfaces":[{"interface":"back","average":{"1min":20.192,"5min":27.117000000000001,"15min":25.914000000000001},"min":{"1min":8.4700000000000006,"5min":8.2810000000000006,"15min":7.5679999999999996},"max":{"1min":63.576999999999998,"5min":488.01299999999998,"15min":488.01299999999998},"last":25.760000000000002},{"interface":"front","average":{"1min":18.07,"5min":21.914999999999999,"15min":26.193999999999999},"min":{"1min":9.5719999999999992,"5min":8.1890000000000001,"15min":8.1890000000000001},"max":{"1min":32.356000000000002,"5min":197.38,"15min":333.43900000000002},"last":13.863}]},{"osd":7,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":23.504000000000001,"5min":30.591999999999999,"15min":25.545000000000002},"min":{"1min":10.676,"5min":7.1340000000000003,"15min":7.1340000000000003},"max":{"1min":100.84399999999999,"5min":322.54599999999999,"15min":322.54599999999999},"last":8.3339999999999996},{"interface":"front","average":{"1min":24.988,"5min":29.010999999999999,"15min":25.050000000000001},"min":{"1min":13.025,"5min":9.6940000000000008,"15min":8.6699999999999999},"max":{"1min":90.594999999999999,"5min":218.30199999999999,"15min":218.30199999999999},"last":15.365}]}]},{"osd":5,"up_from":18,"seq":77309411773,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72600,"kb_used_data":5316,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299240,"statfs":{"total":96636764160,"available":96562421760,"internally_reserved":0,"allocated":5443584,"data_stored":2701160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":9.1660000000000004,"5min":22.483000000000001,"15min":20.535},"min":{"1min":4.9420000000000002,"5min":4.6539999999999999,"15min":4.0940000000000003},"max":{"1min":28.151,"5min":241.49000000000001,"15min":346.541},"last":7.2160000000000002},{"interface":"front","average":{"1min":13.221,"5min":26.934000000000001,"15min":26.605},"min":{"1min":7.6550000000000002,"5min":6.1500000000000004,"15min":5.8819999999999997},"max":{"1min":24.053000000000001,"5min":236.81100000000001,"15min":524.904},"last":12.234999999999999}]},{"osd":1,"last update":"Sun Jan 21 03:55:22 2024","interfaces":[{"interface":"back","average":{"1min":14.785,"5min":26.623999999999999,"15min":20.832999999999998},"min":{"1min":6.2729999999999997,"5min":5.1429999999999998,"15min":4.907},"max":{"1min":84.840999999999994,"5min":284.07400000000001,"15min":368.12400000000002},"last":8.4740000000000002},{"interface":"front","average":{"1min":20.699999999999999,"5min":34.825000000000003,"15min":27.59},"min":{"1min":11.284000000000001,"5min":8.8569999999999993,"15min":7.5359999999999996},"max":{"1min":85.671000000000006,"5min":296.56400000000002,"15min":368.86099999999999},"last":10.801}]},{"osd":2,"last update":"Sun Jan 21 03:55:35 2024","interfaces":[{"interface":"back","average":{"1min":11.273999999999999,"5min":46.090000000000003,"15min":29.077000000000002},"min":{"1min":7.25,"5min":6.6310000000000002,"15min":6.0579999999999998},"max":{"1min":27.489000000000001,"5min":2166.2739999999999,"15min":2166.2739999999999},"last":10.039999999999999},{"interface":"front","average":{"1min":19.923999999999999,"5min":53.329000000000001,"15min":37.494},"min":{"1min":9.8030000000000008,"5min":7.1150000000000002,"15min":7.1150000000000002},"max":{"1min":71.177999999999997,"5min":2181.6190000000001,"15min":2181.6190000000001},"last":15.766}]},{"osd":3,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":16.638000000000002,"5min":30.282,"15min":26.837},"min":{"1min":6.8179999999999996,"5min":6.2770000000000001,"15min":6.2770000000000001},"max":{"1min":63.612000000000002,"5min":412.714,"15min":412.714},"last":16.672999999999998},{"interface":"front","average":{"1min":20.707999999999998,"5min":34.683999999999997,"15min":28.085999999999999},"min":{"1min":9.9399999999999995,"5min":9.6389999999999993,"15min":6.8840000000000003},"max":{"1min":57.048000000000002,"5min":314.28199999999998,"15min":314.28199999999998},"last":13.682}]},{"osd":4,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":21.478000000000002,"5min":39.764000000000003,"15min":27.739999999999998},"min":{"1min":12.226000000000001,"5min":7.7060000000000004,"15min":6.274},"max":{"1min":49.774999999999999,"5min":418.18299999999999,"15min":418.18299999999999},"last":18.27},{"interface":"front","average":{"1min":24.219000000000001,"5min":42.651000000000003,"15min":30.201000000000001},"min":{"1min":14.734999999999999,"5min":11.993,"15min":7.5490000000000004},"max":{"1min":57.712000000000003,"5min":387.125,"15min":387.125},"last":19.117999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":17.492999999999999,"5min":32.762,"15min":25.588999999999999},"min":{"1min":8.6739999999999995,"5min":6.7050000000000001,"15min":6.7050000000000001},"max":{"1min":31.645,"5min":431.40899999999999,"15min":431.40899999999999},"last":11.577},{"interface":"front","average":{"1min":23.259,"5min":36.051000000000002,"15min":28.965},"min":{"1min":11.241,"5min":9.0809999999999995,"15min":8.5470000000000006},"max":{"1min":61.225999999999999,"5min":432.19,"15min":432.19},"last":13.022}]},{"osd":7,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":23.452000000000002,"5min":38.652000000000001,"15min":29.713000000000001},"min":{"1min":13.459,"5min":9.7560000000000002,"15min":7.8639999999999999},"max":{"1min":48.789999999999999,"5min":281.06099999999998,"15min":281.06099999999998},"last":17.498999999999999},{"interface":"front","average":{"1min":12.581,"5min":26.823,"15min":22.974},"min":{"1min":6.5170000000000003,"5min":6.5170000000000003,"15min":6.5170000000000003},"max":{"1min":36.563000000000002,"5min":282.29000000000002,"15min":282.29000000000002},"last":9.2140000000000004}]}]},{"osd":7,"up_from":14,"seq":60129542586,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72224,"kb_used_data":4944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299616,"statfs":{"total":96636764160,"available":96562806784,"internally_reserved":0,"allocated":5062656,"data_stored":2314831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:17 2024","interfaces":[{"interface":"back","average":{"1min":11.395,"5min":20.713000000000001,"15min":16.225999999999999},"min":{"1min":5.0099999999999998,"5min":4.7450000000000001,"15min":4.1929999999999996},"max":{"1min":27.079000000000001,"5min":239.37799999999999,"15min":304.20800000000003},"last":26.015999999999998},{"interface":"front","average":{"1min":10.247999999999999,"5min":21.539000000000001,"15min":16.581},"min":{"1min":5.9039999999999999,"5min":5.2709999999999999,"15min":4.2679999999999998},"max":{"1min":31.059999999999999,"5min":181.04599999999999,"15min":181.04599999999999},"last":22.859000000000002}]},{"osd":1,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":12.342000000000001,"5min":19.533000000000001,"15min":20.614000000000001},"min":{"1min":6.0869999999999997,"5min":5.6580000000000004,"15min":5.0419999999999998},"max":{"1min":27.402999999999999,"5min":138.82499999999999,"15min":248.17500000000001},"last":13.691000000000001},{"interface":"front","average":{"1min":14.029,"5min":23.824999999999999,"15min":23.670999999999999},"min":{"1min":7.9109999999999996,"5min":6.3769999999999998,"15min":6.3769999999999998},"max":{"1min":26.454000000000001,"5min":214.55000000000001,"15min":277.85700000000003},"last":11.555}]},{"osd":2,"last update":"Sun Jan 21 03:56:17 2024","interfaces":[{"interface":"back","average":{"1min":17.594999999999999,"5min":36.784999999999997,"15min":27.683},"min":{"1min":7.2869999999999999,"5min":7.2290000000000001,"15min":6.1779999999999999},"max":{"1min":47.972999999999999,"5min":683.21000000000004,"15min":683.21000000000004},"last":18.545000000000002},{"interface":"front","average":{"1min":16.43,"5min":24.434999999999999,"15min":22.135999999999999},"min":{"1min":9.0399999999999991,"5min":6.3890000000000002,"15min":6.3890000000000002},"max":{"1min":35.869,"5min":150.369,"15min":367.30000000000001},"last":15.68}]},{"osd":3,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":17.163,"5min":27.100000000000001,"15min":22.632000000000001},"min":{"1min":10.734999999999999,"5min":7.1470000000000002,"15min":6.8040000000000003},"max":{"1min":24.895,"5min":192.14599999999999,"15min":192.14599999999999},"last":12.595000000000001},{"interface":"front","average":{"1min":16.158000000000001,"5min":24.597000000000001,"15min":21.649000000000001},"min":{"1min":7.851,"5min":7.8090000000000002,"15min":6.5960000000000001},"max":{"1min":29.548999999999999,"5min":166.846,"15min":210.006},"last":9.8450000000000006}]},{"osd":4,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":20.638999999999999,"5min":27.516999999999999,"15min":22.658999999999999},"min":{"1min":9.4700000000000006,"5min":7.6260000000000003,"15min":7.2309999999999999},"max":{"1min":43.503,"5min":197.31,"15min":268.416},"last":30.398},{"interface":"front","average":{"1min":17.760999999999999,"5min":25.548999999999999,"15min":21.73},"min":{"1min":7.516,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":33.429000000000002,"5min":143.07400000000001,"15min":168.95599999999999},"last":28.981999999999999}]},{"osd":5,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":17.228999999999999,"5min":25.649999999999999,"15min":24.148},"min":{"1min":7.3029999999999999,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":25.670000000000002,"5min":149.56,"15min":226.947},"last":20.795999999999999},{"interface":"front","average":{"1min":17.081,"5min":24.050000000000001,"15min":23.798999999999999},"min":{"1min":8.6419999999999995,"5min":7.5839999999999996,"15min":7.5839999999999996},"max":{"1min":41.097000000000001,"5min":137.00200000000001,"15min":248.94800000000001},"last":28.298999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":21.542000000000002,"5min":27.488,"15min":26.065999999999999},"min":{"1min":10.897,"5min":9.4459999999999997,"15min":9.4459999999999997},"max":{"1min":55.081000000000003,"5min":193.16,"15min":295.12099999999998},"last":27.280000000000001},{"interface":"front","average":{"1min":19.73,"5min":28.16,"15min":26.513999999999999},"min":{"1min":11.029,"5min":9.3219999999999992,"15min":9.3219999999999992},"max":{"1min":55.758000000000003,"5min":178.01900000000001,"15min":278.71899999999999},"last":23.907}]}]},{"osd":4,"up_from":14,"seq":60129542585,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72220,"kb_used_data":4936,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299620,"statfs":{"total":96636764160,"available":96562810880,"internally_reserved":0,"allocated":5054464,"data_stored":2310730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":8.7889999999999997,"5min":15.209,"15min":16.693000000000001},"min":{"1min":4.9939999999999998,"5min":4.4660000000000002,"15min":4.4100000000000001},"max":{"1min":18.459,"5min":220.72900000000001,"15min":220.72900000000001},"last":5.0410000000000004},{"interface":"front","average":{"1min":12.965,"5min":16.300000000000001,"15min":18.149999999999999},"min":{"1min":6.6360000000000001,"5min":4.6029999999999998,"15min":4.6029999999999998},"max":{"1min":45.616,"5min":101.768,"15min":204.142},"last":23.486999999999998}]},{"osd":1,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":12.475,"5min":22.411999999999999,"15min":18.585000000000001},"min":{"1min":6.6440000000000001,"5min":4.9359999999999999,"15min":4.9359999999999999},"max":{"1min":26.292999999999999,"5min":272.61399999999998,"15min":272.61399999999998},"last":20.783999999999999},{"interface":"front","average":{"1min":16.969000000000001,"5min":24.193000000000001,"15min":20.829000000000001},"min":{"1min":9.1150000000000002,"5min":6.8719999999999999,"15min":5.6200000000000001},"max":{"1min":49.414999999999999,"5min":142.97300000000001,"15min":223.49299999999999},"last":34.969000000000001}]},{"osd":2,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":23.027000000000001,"5min":29.632000000000001,"15min":24.187999999999999},"min":{"1min":9.3979999999999997,"5min":7.4320000000000004,"15min":6.6879999999999997},"max":{"1min":77.858999999999995,"5min":285.61000000000001,"15min":285.61000000000001},"last":25.93},{"interface":"front","average":{"1min":21.07,"5min":26.555,"15min":22.710999999999999},"min":{"1min":7.6769999999999996,"5min":6.899,"15min":6.2080000000000002},"max":{"1min":101.676,"5min":298.892,"15min":298.892},"last":8.9760000000000009}]},{"osd":3,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":14.119,"5min":19.032,"15min":20.623999999999999},"min":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"max":{"1min":48.497,"5min":132.958,"15min":200.697},"last":29.013999999999999},{"interface":"front","average":{"1min":19.265000000000001,"5min":23.931999999999999,"15min":21.149000000000001},"min":{"1min":8.9890000000000008,"5min":7.5250000000000004,"15min":7.1539999999999999},"max":{"1min":60.786000000000001,"5min":166.66200000000001,"15min":188.06399999999999},"last":24.388000000000002}]},{"osd":5,"last update":"Sun Jan 21 03:55:58 2024","interfaces":[{"interface":"back","average":{"1min":17.315000000000001,"5min":26.736000000000001,"15min":24.969000000000001},"min":{"1min":10.885999999999999,"5min":8.2970000000000006,"15min":6.7359999999999998},"max":{"1min":30.718,"5min":182.53700000000001,"15min":470.87700000000001},"last":19.791},{"interface":"front","average":{"1min":13.831,"5min":23.012,"15min":24.087},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":6.944},"max":{"1min":23.405000000000001,"5min":145.125,"15min":471.56999999999999},"last":26.879999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:55:37 2024","interfaces":[{"interface":"back","average":{"1min":21.370999999999999,"5min":25.059999999999999,"15min":21.741},"min":{"1min":14.724,"5min":8.5820000000000007,"15min":6.415},"max":{"1min":73.459999999999994,"5min":208.42099999999999,"15min":285.87},"last":30.898},{"interface":"front","average":{"1min":18.635000000000002,"5min":26.960000000000001,"15min":24.614000000000001},"min":{"1min":9.5129999999999999,"5min":9.5129999999999999,"15min":7.5430000000000001},"max":{"1min":64.510000000000005,"5min":252.47999999999999,"15min":252.47999999999999},"last":25.187999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:55:58 2024","interfaces":[{"interface":"back","average":{"1min":20.036999999999999,"5min":25.931000000000001,"15min":22.280000000000001},"min":{"1min":12.244,"5min":9.3040000000000003,"15min":7.1820000000000004},"max":{"1min":44.164999999999999,"5min":141.51300000000001,"15min":229.04900000000001},"last":28.405000000000001},{"interface":"front","average":{"1min":22.382999999999999,"5min":28.206,"15min":22.698},"min":{"1min":15.759,"5min":12.417999999999999,"15min":7.4299999999999997},"max":{"1min":42.031999999999996,"5min":179.12799999999999,"15min":179.12799999999999},"last":31.728000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T03:56:26.873 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-21T03:56:26.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T03:56:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.979+0000 7fa7a9846640 10 monclient: get_monmap_and_config 2024-01-21T03:56:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.979+0000 7fa7a9846640 10 monclient: build_initial_monmap 2024-01-21T03:56:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.979+0000 7fa7a9846640 10 monclient: monmap: 2024-01-21T03:56:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:26.980006+0000 2024-01-21T03:56:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:26.980006+0000 2024-01-21T03:56:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:26.981 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.979+0000 7fa7a9846640 1 Processor -- start 2024-01-21T03:56:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.979+0000 7fa7a9846640 1 -- start start 2024-01-21T03:56:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.979+0000 7fa7a9846640 10 monclient: init 2024-01-21T03:56:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-01-21T03:56:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7a40a8a40 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa7a40b21c0 0x7fa7a413f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a40b21c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7a40a8e20 0x7fa7a413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a40a8e20 addr v2:172.21.15.7:3300/0 2024-01-21T03:56:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a40aa9b0 con 0x7fa7a40a8a40 2024-01-21T03:56:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a40aa7d0 con 0x7fa7a40a8e20 2024-01-21T03:56:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a40aac90 con 0x7fa7a40b21c0 2024-01-21T03:56:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.980+0000 7fa7a9846640 10 monclient(hunting): authenticate will time out at 3533.296071s 2024-01-21T03:56:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.983+0000 7fa7a27fc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa7a40b21c0 0x7fa7a413f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:26.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.984+0000 7fa7a37fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7a40a8e20 0x7fa7a413f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.984+0000 7fa7a27fc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa7a40b21c0 0x7fa7a413f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:36962/0 (socket says 172.21.15.7:36962) 2024-01-21T03:56:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.984+0000 7fa7a27fc640 1 -- 172.21.15.7:0/476331234 learned_addr learned my addr 172.21.15.7:0/476331234 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.984+0000 7fa7a27fc640 10 monclient(hunting): get_auth_request con 0x7fa7a40b21c0 auth_method 0 2024-01-21T03:56:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.984+0000 7fa7a27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.984+0000 7fa7a27fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.984+0000 7fa7a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.985+0000 7fa7a37fe640 10 monclient(hunting): get_auth_request con 0x7fa7a40a8e20 auth_method 0 2024-01-21T03:56:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.985+0000 7fa7a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.985+0000 7fa7a37fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.985+0000 7fa7a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.987+0000 7fa7a27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.987+0000 7fa7a27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.987+0000 7fa7a27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:26.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.987+0000 7fa7a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:26.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.987+0000 7fa7a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:26.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.987+0000 7fa7a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:26.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.995+0000 7fa7a37fe640 10 monclient(hunting): handle_auth_done global_id 15831 payload 274 2024-01-21T03:56:26.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.995+0000 7fa7a37fe640 1 -- 172.21.15.7:0/476331234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa7a40b21c0 msgr2=0x7fa7a413f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:26.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.995+0000 7fa7a37fe640 1 --2- 172.21.15.7:0/476331234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa7a40b21c0 0x7fa7a413f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:26.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.995+0000 7fa7a37fe640 1 -- 172.21.15.7:0/476331234 >> v1:172.21.15.177:6789/0 conn(0x7fa7a40a8a40 legacy=0x7fa7a4145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:26.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.995+0000 7fa7a37fe640 10 monclient: _finish_hunting 0 2024-01-21T03:56:26.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.995+0000 7fa7a37fe640 1 monclient: found mon.noname-a 2024-01-21T03:56:26.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.995+0000 7fa7a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:26.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.995+0000 7fa7a37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:56:26.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:26.995+0000 7fa7a37fe640 1 -- 172.21.15.7:0/476331234 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a40aaf70 con 0x7fa7a40a8e20 2024-01-21T03:56:27.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.001+0000 7fa7a37fe640 1 --2- 172.21.15.7:0/476331234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a40a8e20 0x7fa7a413f6e0 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7fa79c009d90 tx=0x7fa79c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff89a6c24e2393fa server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:27.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a1ffb640 1 -- 172.21.15.7:0/476331234 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c02cc50 con 0x7fa7a40a8e20 2024-01-21T03:56:27.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:27.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:56:27.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a1ffb640 10 monclient: dump: 2024-01-21T03:56:27.017 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:27.017 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:27.018 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a1ffb640 10 monclient: _finish_auth 0 2024-01-21T03:56:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a1ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:56:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a1ffb640 1 -- 172.21.15.7:0/476331234 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79c02cdb0 con 0x7fa7a40a8e20 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a1ffb640 1 -- 172.21.15.7:0/476331234 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c034660 con 0x7fa7a40a8e20 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a9846640 5 monclient: authenticate success, global_id 15831 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a9846640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a9846640 4 set_mon_vals no callback set 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.016+0000 7fa7a1ffb640 10 monclient: dump: 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:27.019 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:27.020 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:27.020 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:27.020 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:27.020 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.017+0000 7fa7a9846640 10 monclient: shutdown 2024-01-21T03:56:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.017+0000 7fa7a9846640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.017+0000 7fa7a9846640 1 -- 172.21.15.7:0/476331234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a40a8e20 msgr2=0x7fa7a413f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.017+0000 7fa7a9846640 1 --2- 172.21.15.7:0/476331234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a40a8e20 0x7fa7a413f6e0 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7fa79c009d90 tx=0x7fa79c02bea0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.017+0000 7fa7a9846640 1 -- 172.21.15.7:0/476331234 shutdown_connections 2024-01-21T03:56:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.017+0000 7fa7a9846640 1 --2- 172.21.15.7:0/476331234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a40a8e20 0x7fa7a413f6e0 unknown :-1 s=CLOSED pgs=3085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.017+0000 7fa7a9846640 1 --2- 172.21.15.7:0/476331234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa7a40b21c0 0x7fa7a413f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.017+0000 7fa7a9846640 1 -- 172.21.15.7:0/476331234 >> 172.21.15.7:0/476331234 conn(0x7fa7a4066f50 msgr2=0x7fa7a40b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.017+0000 7fa7a9846640 1 -- 172.21.15.7:0/476331234 shutdown_connections 2024-01-21T03:56:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.017+0000 7fa7a9846640 1 -- 172.21.15.7:0/476331234 wait complete. 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.018+0000 7fa7a9846640 10 monclient: build_initial_monmap 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.018+0000 7fa7a9846640 10 monclient: monmap: 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.018+0000 7fa7a9846640 1 librados: starting msgr at 2024-01-21T03:56:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.018+0000 7fa7a9846640 1 librados: starting objecter 2024-01-21T03:56:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.018+0000 7fa7a9846640 1 Processor -- start 2024-01-21T03:56:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.018+0000 7fa7a9846640 1 -- start start 2024-01-21T03:56:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.018+0000 7fa7a9846640 1 librados: setting wanted keys 2024-01-21T03:56:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.018+0000 7fa7a9846640 1 librados: calling monclient init 2024-01-21T03:56:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.018+0000 7fa7a9846640 10 monclient: init 2024-01-21T03:56:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T03:56:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa7a40a8a40 0x7fa7a4139e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a40a8a40 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a40a8e20 0x7fa7a4141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a40a8e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7a40b21c0 0x7fa7a41d7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a40b21c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a41426d0 con 0x7fa7a40a8e20 2024-01-21T03:56:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a40b5df0 con 0x7fa7a40b21c0 2024-01-21T03:56:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a4142100 con 0x7fa7a40a8a40 2024-01-21T03:56:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.019+0000 7fa7a9846640 10 monclient(hunting): authenticate will time out at 3533.335119s 2024-01-21T03:56:27.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.027+0000 7fa7a2ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa7a40a8a40 0x7fa7a4139e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.027+0000 7fa7a27fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a40a8e20 0x7fa7a4141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.028+0000 7fa7a2ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa7a40a8a40 0x7fa7a4139e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:36966/0 (socket says 172.21.15.7:36966) 2024-01-21T03:56:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.028+0000 7fa7a2ffd640 1 -- 172.21.15.7:0/2157138376 learned_addr learned my addr 172.21.15.7:0/2157138376 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.028+0000 7fa7a2ffd640 10 monclient(hunting): get_auth_request con 0x7fa7a40a8a40 auth_method 0 2024-01-21T03:56:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.028+0000 7fa7a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.028+0000 7fa7a2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.028+0000 7fa7a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.028+0000 7fa7a27fc640 10 monclient(hunting): get_auth_request con 0x7fa7a40a8e20 auth_method 0 2024-01-21T03:56:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.028+0000 7fa7a27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.028+0000 7fa7a27fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.028+0000 7fa7a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:27.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.030+0000 7fa7a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:27.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.030+0000 7fa7a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:27.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.030+0000 7fa7a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.033+0000 7fa7a27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.033+0000 7fa7a27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.033+0000 7fa7a27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:27.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.038+0000 7fa7a27fc640 10 monclient(hunting): handle_auth_done global_id 15834 payload 824 2024-01-21T03:56:27.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.038+0000 7fa7a27fc640 1 -- 172.21.15.7:0/2157138376 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa7a40a8a40 msgr2=0x7fa7a4139e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:27.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.038+0000 7fa7a27fc640 1 --2- 172.21.15.7:0/2157138376 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa7a40a8a40 0x7fa7a4139e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.038+0000 7fa7a27fc640 1 -- 172.21.15.7:0/2157138376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7a40b21c0 msgr2=0x7fa7a41d7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:27.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.038+0000 7fa7a27fc640 1 --2- 172.21.15.7:0/2157138376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7a40b21c0 0x7fa7a41d7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.038+0000 7fa7a27fc640 10 monclient: _finish_hunting 0 2024-01-21T03:56:27.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.038+0000 7fa7a27fc640 1 monclient: found mon.noname-a 2024-01-21T03:56:27.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.038+0000 7fa7a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:27.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.039+0000 7fa7a27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:56:27.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.039+0000 7fa7a27fc640 1 -- 172.21.15.7:0/2157138376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa79c009a70 con 0x7fa7a40a8e20 2024-01-21T03:56:27.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.058+0000 7fa7a27fc640 1 --2- 172.21.15.7:0/2157138376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a40a8e20 0x7fa7a4141bc0 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7fa79800e8a0 tx=0x7fa79800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:27.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.058+0000 7fa78bfff640 10 client.?.objecter ms_handle_connect 0x7fa7a40a8e20 2024-01-21T03:56:27.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.058+0000 7fa78bfff640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:27.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.077+0000 7fa78bfff640 1 -- 172.21.15.7:0/2157138376 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa798009e60 con 0x7fa7a40a8e20 2024-01-21T03:56:27.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa78bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:27.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa78bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:56:27.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa78bfff640 10 monclient: dump: 2024-01-21T03:56:27.079 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:27.079 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:27.079 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.079 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.079 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:27.079 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:27.079 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:27.079 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa78bfff640 10 monclient: _finish_auth 0 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa78bfff640 10 monclient: _check_auth_tickets 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa78bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa78bfff640 1 -- 172.21.15.7:0/2157138376 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79800c940 con 0x7fa7a40a8e20 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa78bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa78bfff640 1 -- 172.21.15.7:0/2157138376 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79800ccd0 con 0x7fa7a40a8e20 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa7a9846640 5 monclient: authenticate success, global_id 15834 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa78bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa78bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa78bfff640 10 monclient: dump: 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:27.080 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa7a9846640 10 monclient: _renew_subs 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa7a9846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa7a17fa640 4 set_mon_vals no callback set 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa7a9846640 1 -- 172.21.15.7:0/2157138376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a41423e0 con 0x7fa7a40a8e20 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa7a9846640 10 client.15834.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa7a9846640 10 monclient: _renew_subs 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa7a9846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa7a9846640 1 -- 172.21.15.7:0/2157138376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a41d78b0 con 0x7fa7a40a8e20 2024-01-21T03:56:27.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.078+0000 7fa7a9846640 1 librados: init done 2024-01-21T03:56:27.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.079+0000 7fa7a9846640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:27.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.079+0000 7fa7a9846640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:27.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.079+0000 7fa7a9846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:27.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.079+0000 7fa7a9846640 1 -- 172.21.15.7:0/2157138376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a4142880 con 0x7fa7a40a8e20 2024-01-21T03:56:27.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.351+0000 7fa78bfff640 1 -- 172.21.15.7:0/2157138376 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa798013470 con 0x7fa7a40a8e20 2024-01-21T03:56:27.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.351+0000 7fa78bfff640 1 --2- 172.21.15.7:0/2157138376 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fa79006c8b0 0x7fa79006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:27.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.352+0000 7fa7a2ffd640 1 --2- 172.21.15.7:0/2157138376 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fa79006c8b0 0x7fa79006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:27.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.352+0000 7fa7a2ffd640 10 monclient: get_auth_request con 0x7fa79006c8b0 auth_method 0 2024-01-21T03:56:27.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.352+0000 7fa7a2ffd640 1 --2- 172.21.15.7:0/2157138376 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fa79006c8b0 0x7fa79006ed70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa7940018b0 tx=0x7fa794009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:27.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.352+0000 7fa78bfff640 10 client.15834.objecter ms_handle_connect 0x7fa79006c8b0 2024-01-21T03:56:27.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.366+0000 7fa78bfff640 1 -- 172.21.15.7:0/2157138376 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7fa798058820 con 0x7fa7a40a8e20 2024-01-21T03:56:27.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.366+0000 7fa78bfff640 10 client.15834.objecter ms_dispatch 0x7fa7a4000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:27.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.366+0000 7fa78bfff640 3 client.15834.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:27.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.366+0000 7fa78bfff640 3 client.15834.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:27.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.366+0000 7fa78bfff640 20 client.15834.objecter dump_active .. 0 homeless 2024-01-21T03:56:27.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.366+0000 7fa78bfff640 1 -- 172.21.15.7:0/2157138376 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa798054b60 con 0x7fa7a40a8e20 2024-01-21T03:56:27.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.366+0000 7fa78bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:27.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.366+0000 7fa78bfff640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:27.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.548+0000 7fa7a9846640 1 -- 172.21.15.7:0/2157138376 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa7a4142880 con 0x7fa79006c8b0 2024-01-21T03:56:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.553+0000 7fa78bfff640 1 -- 172.21.15.7:0/2157138376 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151878 (secure 0 0 0) 0x7fa7a4142880 con 0x7fa79006c8b0 2024-01-21T03:56:27.555 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:56:27.556 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-01-21T03:56:27.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 10 librados: watch_flush enter 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 10 librados: watch_flush exit 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 20 client.15834.objecter shutdown clearing up homeless session... 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 10 client.15834.objecter successfully canceled tick 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 1 -- 172.21.15.7:0/2157138376 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fa79006c8b0 msgr2=0x7fa79006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 1 --2- 172.21.15.7:0/2157138376 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fa79006c8b0 0x7fa79006ed70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa7940018b0 tx=0x7fa794009040 comp rx=0 tx=0).stop 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 10 monclient: shutdown 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 1 -- 172.21.15.7:0/2157138376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a40a8e20 msgr2=0x7fa7a4141bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 1 --2- 172.21.15.7:0/2157138376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a40a8e20 0x7fa7a4141bc0 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7fa79800e8a0 tx=0x7fa79800ed60 comp rx=0 tx=0).stop 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 1 -- 172.21.15.7:0/2157138376 shutdown_connections 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 1 --2- 172.21.15.7:0/2157138376 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fa79006c8b0 0x7fa79006ed70 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 1 --2- 172.21.15.7:0/2157138376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7a40b21c0 0x7fa7a41d7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 1 --2- 172.21.15.7:0/2157138376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a40a8e20 0x7fa7a4141bc0 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 1 --2- 172.21.15.7:0/2157138376 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa7a40a8a40 0x7fa7a4139e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.555+0000 7fa7a9846640 1 -- 172.21.15.7:0/2157138376 >> 172.21.15.7:0/2157138376 conn(0x7fa7a4066f50 msgr2=0x7fa7a41369e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:27.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.556+0000 7fa7a9846640 1 -- 172.21.15.7:0/2157138376 shutdown_connections 2024-01-21T03:56:27.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.556+0000 7fa7a9846640 1 -- 172.21.15.7:0/2157138376 wait complete. 2024-01-21T03:56:27.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.556+0000 7fa7a9846640 1 librados: shutdown 2024-01-21T03:56:27.568 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":2398,"stamp":"2024-01-21T03:56:26.232835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":600688,"kb_used_data":40292,"kb_used_omap":12,"kb_used_meta":560243,"kb_avail":754374032,"statfs":{"total":773094113280,"available":772479008768,"internally_reserved":0,"allocated":41259008,"data_stored":19284616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":573689393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":146,"apply_latency_ms":146,"commit_latency_ns":146000000,"apply_latency_ns":146000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.330366"},"pg_stats":[{"pgid":"14.1e","version":"0'0","reported_seq":1254,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.191591+0000","last_change":"2024-01-21T03:21:15.540562+0000","last_active":"2024-01-21T03:56:23.191591+0000","last_peered":"2024-01-21T03:56:23.191591+0000","last_clean":"2024-01-21T03:56:23.191591+0000","last_became_active":"2024-01-21T03:21:15.289479+0000","last_became_peered":"2024-01-21T03:21:15.289479+0000","last_unstale":"2024-01-21T03:56:23.191591+0000","last_undegraded":"2024-01-21T03:56:23.191591+0000","last_fullsized":"2024-01-21T03:56:23.191591+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:26:47.504093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1210,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.683250+0000","last_change":"2024-01-21T03:21:15.879415+0000","last_active":"2024-01-21T03:56:20.683250+0000","last_peered":"2024-01-21T03:56:20.683250+0000","last_clean":"2024-01-21T03:56:20.683250+0000","last_became_active":"2024-01-21T03:21:15.864718+0000","last_became_peered":"2024-01-21T03:21:15.864718+0000","last_unstale":"2024-01-21T03:56:20.683250+0000","last_undegraded":"2024-01-21T03:56:20.683250+0000","last_fullsized":"2024-01-21T03:56:20.683250+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:26:29.398180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1231,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.706988+0000","last_change":"2024-01-21T03:21:09.350719+0000","last_active":"2024-01-21T03:56:19.706988+0000","last_peered":"2024-01-21T03:56:19.706988+0000","last_clean":"2024-01-21T03:56:19.706988+0000","last_became_active":"2024-01-21T03:21:09.333873+0000","last_became_peered":"2024-01-21T03:21:09.333873+0000","last_unstale":"2024-01-21T03:56:19.706988+0000","last_undegraded":"2024-01-21T03:56:19.706988+0000","last_fullsized":"2024-01-21T03:56:19.706988+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:44:56.009576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1230,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.697115+0000","last_change":"2024-01-21T03:21:13.545831+0000","last_active":"2024-01-21T03:56:20.697115+0000","last_peered":"2024-01-21T03:56:20.697115+0000","last_clean":"2024-01-21T03:56:20.697115+0000","last_became_active":"2024-01-21T03:21:13.539461+0000","last_became_peered":"2024-01-21T03:21:13.539461+0000","last_unstale":"2024-01-21T03:56:20.697115+0000","last_undegraded":"2024-01-21T03:56:20.697115+0000","last_fullsized":"2024-01-21T03:56:20.697115+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:55:11.327991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1207,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.484590+0000","last_change":"2024-01-21T03:21:31.051184+0000","last_active":"2024-01-21T03:56:19.484590+0000","last_peered":"2024-01-21T03:56:19.484590+0000","last_clean":"2024-01-21T03:56:19.484590+0000","last_became_active":"2024-01-21T03:21:30.818768+0000","last_became_peered":"2024-01-21T03:21:30.818768+0000","last_unstale":"2024-01-21T03:56:19.484590+0000","last_undegraded":"2024-01-21T03:56:19.484590+0000","last_fullsized":"2024-01-21T03:56:19.484590+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:16:32.050608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1232,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.685544+0000","last_change":"2024-01-21T03:21:13.851032+0000","last_active":"2024-01-21T03:56:20.685544+0000","last_peered":"2024-01-21T03:56:20.685544+0000","last_clean":"2024-01-21T03:56:20.685544+0000","last_became_active":"2024-01-21T03:21:13.842398+0000","last_became_peered":"2024-01-21T03:21:13.842398+0000","last_unstale":"2024-01-21T03:56:20.685544+0000","last_undegraded":"2024-01-21T03:56:20.685544+0000","last_fullsized":"2024-01-21T03:56:20.685544+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:27:03.441063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1233,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.669592+0000","last_change":"2024-01-21T03:21:34.296238+0000","last_active":"2024-01-21T03:56:20.669592+0000","last_peered":"2024-01-21T03:56:20.669592+0000","last_clean":"2024-01-21T03:56:20.669592+0000","last_became_active":"2024-01-21T03:21:34.290896+0000","last_became_peered":"2024-01-21T03:21:34.290896+0000","last_unstale":"2024-01-21T03:56:20.669592+0000","last_undegraded":"2024-01-21T03:56:20.669592+0000","last_fullsized":"2024-01-21T03:56:20.669592+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:52:43.200440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1","version":"60'7","reported_seq":1224,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.681000+0000","last_change":"2024-01-21T03:21:35.777407+0000","last_active":"2024-01-21T03:56:20.681000+0000","last_peered":"2024-01-21T03:56:20.681000+0000","last_clean":"2024-01-21T03:56:20.681000+0000","last_became_active":"2024-01-21T03:21:35.763549+0000","last_became_peered":"2024-01-21T03:21:35.763549+0000","last_unstale":"2024-01-21T03:56:20.681000+0000","last_undegraded":"2024-01-21T03:56:20.681000+0000","last_fullsized":"2024-01-21T03:56:20.681000+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:56:17.661965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1023,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.957634+0000","last_change":"2024-01-21T03:26:06.764781+0000","last_active":"2024-01-21T03:56:24.957634+0000","last_peered":"2024-01-21T03:56:24.957634+0000","last_clean":"2024-01-21T03:56:24.957634+0000","last_became_active":"2024-01-21T03:26:06.758343+0000","last_became_peered":"2024-01-21T03:26:06.758343+0000","last_unstale":"2024-01-21T03:56:24.957634+0000","last_undegraded":"2024-01-21T03:56:24.957634+0000","last_fullsized":"2024-01-21T03:56:24.957634+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:09:42.970888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1226,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.259577+0000","last_change":"2024-01-21T03:21:15.810535+0000","last_active":"2024-01-21T03:56:20.259577+0000","last_peered":"2024-01-21T03:56:20.259577+0000","last_clean":"2024-01-21T03:56:20.259577+0000","last_became_active":"2024-01-21T03:21:15.779865+0000","last_became_peered":"2024-01-21T03:21:15.779865+0000","last_unstale":"2024-01-21T03:56:20.259577+0000","last_undegraded":"2024-01-21T03:56:20.259577+0000","last_fullsized":"2024-01-21T03:56:20.259577+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:56:54.491579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1207,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.502562+0000","last_change":"2024-01-21T03:21:22.661192+0000","last_active":"2024-01-21T03:56:19.502562+0000","last_peered":"2024-01-21T03:56:19.502562+0000","last_clean":"2024-01-21T03:56:19.502562+0000","last_became_active":"2024-01-21T03:21:22.647696+0000","last_became_peered":"2024-01-21T03:21:22.647696+0000","last_unstale":"2024-01-21T03:56:19.502562+0000","last_undegraded":"2024-01-21T03:56:19.502562+0000","last_fullsized":"2024-01-21T03:56:19.502562+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:05:43.481850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1218,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.275498+0000","last_change":"2024-01-21T03:21:26.231500+0000","last_active":"2024-01-21T03:56:20.275498+0000","last_peered":"2024-01-21T03:56:20.275498+0000","last_clean":"2024-01-21T03:56:20.275498+0000","last_became_active":"2024-01-21T03:21:26.223079+0000","last_became_peered":"2024-01-21T03:21:26.223079+0000","last_unstale":"2024-01-21T03:56:20.275498+0000","last_undegraded":"2024-01-21T03:56:20.275498+0000","last_fullsized":"2024-01-21T03:56:20.275498+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:36:33.965862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1232,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.261261+0000","last_change":"2024-01-21T03:26:06.699602+0000","last_active":"2024-01-21T03:56:20.261261+0000","last_peered":"2024-01-21T03:56:20.261261+0000","last_clean":"2024-01-21T03:56:20.261261+0000","last_became_active":"2024-01-21T03:26:06.693281+0000","last_became_peered":"2024-01-21T03:26:06.693281+0000","last_unstale":"2024-01-21T03:56:20.261261+0000","last_undegraded":"2024-01-21T03:56:20.261261+0000","last_fullsized":"2024-01-21T03:56:20.261261+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:29:07.025641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"23'76","reported_seq":1257,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.243138+0000","last_change":"2024-01-21T03:19:59.052204+0000","last_active":"2024-01-21T03:56:20.243138+0000","last_peered":"2024-01-21T03:56:20.243138+0000","last_clean":"2024-01-21T03:56:20.243138+0000","last_became_active":"2024-01-21T03:19:58.060095+0000","last_became_peered":"2024-01-21T03:19:58.060095+0000","last_unstale":"2024-01-21T03:56:20.243138+0000","last_undegraded":"2024-01-21T03:56:20.243138+0000","last_fullsized":"2024-01-21T03:56:20.243138+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:44.383554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:44.383554+0000","last_clean_scrub_stamp":"2024-01-21T03:19:44.383554+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:42:45.445502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1260,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.144254+0000","last_change":"2024-01-21T03:20:02.295222+0000","last_active":"2024-01-21T03:56:23.144254+0000","last_peered":"2024-01-21T03:56:23.144254+0000","last_clean":"2024-01-21T03:56:23.144254+0000","last_became_active":"2024-01-21T03:20:02.289915+0000","last_became_peered":"2024-01-21T03:20:02.289915+0000","last_unstale":"2024-01-21T03:56:23.144254+0000","last_undegraded":"2024-01-21T03:56:23.144254+0000","last_fullsized":"2024-01-21T03:56:23.144254+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:36:11.210361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1218,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.272174+0000","last_change":"2024-01-21T03:21:14.853643+0000","last_active":"2024-01-21T03:56:20.272174+0000","last_peered":"2024-01-21T03:56:20.272174+0000","last_clean":"2024-01-21T03:56:20.272174+0000","last_became_active":"2024-01-21T03:21:14.829810+0000","last_became_peered":"2024-01-21T03:21:14.829810+0000","last_unstale":"2024-01-21T03:56:20.272174+0000","last_undegraded":"2024-01-21T03:56:20.272174+0000","last_fullsized":"2024-01-21T03:56:20.272174+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:30:12.597754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1261,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.105259+0000","last_change":"2024-01-21T03:20:01.812917+0000","last_active":"2024-01-21T03:56:23.105259+0000","last_peered":"2024-01-21T03:56:23.105259+0000","last_clean":"2024-01-21T03:56:23.105259+0000","last_became_active":"2024-01-21T03:20:01.784281+0000","last_became_peered":"2024-01-21T03:20:01.784281+0000","last_unstale":"2024-01-21T03:56:23.105259+0000","last_undegraded":"2024-01-21T03:56:23.105259+0000","last_fullsized":"2024-01-21T03:56:23.105259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:26:56.084862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1254,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.175368+0000","last_change":"2024-01-21T03:21:26.074817+0000","last_active":"2024-01-21T03:56:23.175368+0000","last_peered":"2024-01-21T03:56:23.175368+0000","last_clean":"2024-01-21T03:56:23.175368+0000","last_became_active":"2024-01-21T03:21:26.057996+0000","last_became_peered":"2024-01-21T03:21:26.057996+0000","last_unstale":"2024-01-21T03:56:23.175368+0000","last_undegraded":"2024-01-21T03:56:23.175368+0000","last_fullsized":"2024-01-21T03:56:23.175368+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:22:40.041984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1216,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.674340+0000","last_change":"2024-01-21T03:20:01.846748+0000","last_active":"2024-01-21T03:56:20.674340+0000","last_peered":"2024-01-21T03:56:20.674340+0000","last_clean":"2024-01-21T03:56:20.674340+0000","last_became_active":"2024-01-21T03:20:01.818592+0000","last_became_peered":"2024-01-21T03:20:01.818592+0000","last_unstale":"2024-01-21T03:56:20.674340+0000","last_undegraded":"2024-01-21T03:56:20.674340+0000","last_fullsized":"2024-01-21T03:56:20.674340+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:08:09.861694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1226,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.257372+0000","last_change":"2024-01-21T03:22:06.056192+0000","last_active":"2024-01-21T03:56:20.257372+0000","last_peered":"2024-01-21T03:56:20.257372+0000","last_clean":"2024-01-21T03:56:20.257372+0000","last_became_active":"2024-01-21T03:22:06.040269+0000","last_became_peered":"2024-01-21T03:22:06.040269+0000","last_unstale":"2024-01-21T03:56:20.257372+0000","last_undegraded":"2024-01-21T03:56:20.257372+0000","last_fullsized":"2024-01-21T03:56:20.257372+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:36:24.196035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1260,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.122291+0000","last_change":"2024-01-21T03:20:02.210632+0000","last_active":"2024-01-21T03:56:23.122291+0000","last_peered":"2024-01-21T03:56:23.122291+0000","last_clean":"2024-01-21T03:56:23.122291+0000","last_became_active":"2024-01-21T03:20:02.201314+0000","last_became_peered":"2024-01-21T03:20:02.201314+0000","last_unstale":"2024-01-21T03:56:23.122291+0000","last_undegraded":"2024-01-21T03:56:23.122291+0000","last_fullsized":"2024-01-21T03:56:23.122291+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:55:53.424403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1232,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.668006+0000","last_change":"2024-01-21T03:21:34.314505+0000","last_active":"2024-01-21T03:56:20.668006+0000","last_peered":"2024-01-21T03:56:20.668006+0000","last_clean":"2024-01-21T03:56:20.668006+0000","last_became_active":"2024-01-21T03:21:34.307508+0000","last_became_peered":"2024-01-21T03:21:34.307508+0000","last_unstale":"2024-01-21T03:56:20.668006+0000","last_undegraded":"2024-01-21T03:56:20.668006+0000","last_fullsized":"2024-01-21T03:56:20.668006+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:38:13.727492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1224,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.273664+0000","last_change":"2024-01-21T03:20:02.214415+0000","last_active":"2024-01-21T03:56:20.273664+0000","last_peered":"2024-01-21T03:56:20.273664+0000","last_clean":"2024-01-21T03:56:20.273664+0000","last_became_active":"2024-01-21T03:20:02.207814+0000","last_became_peered":"2024-01-21T03:20:02.207814+0000","last_unstale":"2024-01-21T03:56:20.273664+0000","last_undegraded":"2024-01-21T03:56:20.273664+0000","last_fullsized":"2024-01-21T03:56:20.273664+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:12:28.359349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1215,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.955788+0000","last_change":"2024-01-21T03:21:23.773815+0000","last_active":"2024-01-21T03:56:24.955788+0000","last_peered":"2024-01-21T03:56:24.955788+0000","last_clean":"2024-01-21T03:56:24.955788+0000","last_became_active":"2024-01-21T03:21:23.767027+0000","last_became_peered":"2024-01-21T03:21:23.767027+0000","last_unstale":"2024-01-21T03:56:24.955788+0000","last_undegraded":"2024-01-21T03:56:24.955788+0000","last_fullsized":"2024-01-21T03:56:24.955788+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:56:41.913140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1260,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.171699+0000","last_change":"2024-01-21T03:20:01.878033+0000","last_active":"2024-01-21T03:56:23.171699+0000","last_peered":"2024-01-21T03:56:23.171699+0000","last_clean":"2024-01-21T03:56:23.171699+0000","last_became_active":"2024-01-21T03:20:01.872724+0000","last_became_peered":"2024-01-21T03:20:01.872724+0000","last_unstale":"2024-01-21T03:56:23.171699+0000","last_undegraded":"2024-01-21T03:56:23.171699+0000","last_fullsized":"2024-01-21T03:56:23.171699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:06:41.515884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1207,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.486663+0000","last_change":"2024-01-21T03:21:55.470689+0000","last_active":"2024-01-21T03:56:19.486663+0000","last_peered":"2024-01-21T03:56:19.486663+0000","last_clean":"2024-01-21T03:56:19.486663+0000","last_became_active":"2024-01-21T03:21:55.460120+0000","last_became_peered":"2024-01-21T03:21:55.460120+0000","last_unstale":"2024-01-21T03:56:19.486663+0000","last_undegraded":"2024-01-21T03:56:19.486663+0000","last_fullsized":"2024-01-21T03:56:19.486663+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:03:53.385543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1236,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.701374+0000","last_change":"2024-01-21T03:20:02.260437+0000","last_active":"2024-01-21T03:56:20.701374+0000","last_peered":"2024-01-21T03:56:20.701374+0000","last_clean":"2024-01-21T03:56:20.701374+0000","last_became_active":"2024-01-21T03:20:02.234092+0000","last_became_peered":"2024-01-21T03:20:02.234092+0000","last_unstale":"2024-01-21T03:56:20.701374+0000","last_undegraded":"2024-01-21T03:56:20.701374+0000","last_fullsized":"2024-01-21T03:56:20.701374+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:32:42.605439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1232,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.703783+0000","last_change":"2024-01-21T03:21:09.536570+0000","last_active":"2024-01-21T03:56:19.703783+0000","last_peered":"2024-01-21T03:56:19.703783+0000","last_clean":"2024-01-21T03:56:19.703783+0000","last_became_active":"2024-01-21T03:21:09.531293+0000","last_became_peered":"2024-01-21T03:21:09.531293+0000","last_unstale":"2024-01-21T03:56:19.703783+0000","last_undegraded":"2024-01-21T03:56:19.703783+0000","last_fullsized":"2024-01-21T03:56:19.703783+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:01:17.017524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1256,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.197738+0000","last_change":"2024-01-21T03:20:02.233166+0000","last_active":"2024-01-21T03:56:23.197738+0000","last_peered":"2024-01-21T03:56:23.197738+0000","last_clean":"2024-01-21T03:56:23.197738+0000","last_became_active":"2024-01-21T03:20:02.226975+0000","last_became_peered":"2024-01-21T03:20:02.226975+0000","last_unstale":"2024-01-21T03:56:23.197738+0000","last_undegraded":"2024-01-21T03:56:23.197738+0000","last_fullsized":"2024-01-21T03:56:23.197738+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:36:45.013749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1253,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.119814+0000","last_change":"2024-01-21T03:21:36.938379+0000","last_active":"2024-01-21T03:56:23.119814+0000","last_peered":"2024-01-21T03:56:23.119814+0000","last_clean":"2024-01-21T03:56:23.119814+0000","last_became_active":"2024-01-21T03:21:36.929227+0000","last_became_peered":"2024-01-21T03:21:36.929227+0000","last_unstale":"2024-01-21T03:56:23.119814+0000","last_undegraded":"2024-01-21T03:56:23.119814+0000","last_fullsized":"2024-01-21T03:56:23.119814+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:55:39.626168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1218,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.252018+0000","last_change":"2024-01-21T03:21:13.975490+0000","last_active":"2024-01-21T03:56:20.252018+0000","last_peered":"2024-01-21T03:56:20.252018+0000","last_clean":"2024-01-21T03:56:20.252018+0000","last_became_active":"2024-01-21T03:21:13.969066+0000","last_became_peered":"2024-01-21T03:21:13.969066+0000","last_unstale":"2024-01-21T03:56:20.252018+0000","last_undegraded":"2024-01-21T03:56:20.252018+0000","last_fullsized":"2024-01-21T03:56:20.252018+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:53:18.899854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1231,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.705306+0000","last_change":"2024-01-21T03:21:22.338167+0000","last_active":"2024-01-21T03:56:19.705306+0000","last_peered":"2024-01-21T03:56:19.705306+0000","last_clean":"2024-01-21T03:56:19.705306+0000","last_became_active":"2024-01-21T03:21:22.319120+0000","last_became_peered":"2024-01-21T03:21:22.319120+0000","last_unstale":"2024-01-21T03:56:19.705306+0000","last_undegraded":"2024-01-21T03:56:19.705306+0000","last_fullsized":"2024-01-21T03:56:19.705306+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:35:43.974713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1230,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.681744+0000","last_change":"2024-01-21T03:21:17.587236+0000","last_active":"2024-01-21T03:56:20.681744+0000","last_peered":"2024-01-21T03:56:20.681744+0000","last_clean":"2024-01-21T03:56:20.681744+0000","last_became_active":"2024-01-21T03:21:17.582134+0000","last_became_peered":"2024-01-21T03:21:17.582134+0000","last_unstale":"2024-01-21T03:56:20.681744+0000","last_undegraded":"2024-01-21T03:56:20.681744+0000","last_fullsized":"2024-01-21T03:56:20.681744+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:14:12.050155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1253,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-01-21T03:56:23.167852+0000","last_change":"2024-01-21T03:21:40.581757+0000","last_active":"2024-01-21T03:56:23.167852+0000","last_peered":"2024-01-21T03:56:23.167852+0000","last_clean":"2024-01-21T03:56:23.167852+0000","last_became_active":"2024-01-21T03:21:40.568805+0000","last_became_peered":"2024-01-21T03:21:40.568805+0000","last_unstale":"2024-01-21T03:56:23.167852+0000","last_undegraded":"2024-01-21T03:56:23.167852+0000","last_fullsized":"2024-01-21T03:56:23.167852+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:33:02.221661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1226,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.588370+0000","last_change":"2024-01-21T03:22:06.157185+0000","last_active":"2024-01-21T03:56:20.588370+0000","last_peered":"2024-01-21T03:56:20.588370+0000","last_clean":"2024-01-21T03:56:20.588370+0000","last_became_active":"2024-01-21T03:22:06.149457+0000","last_became_peered":"2024-01-21T03:22:06.149457+0000","last_unstale":"2024-01-21T03:56:20.588370+0000","last_undegraded":"2024-01-21T03:56:20.588370+0000","last_fullsized":"2024-01-21T03:56:20.588370+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:47:38.535776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1230,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.721033+0000","last_change":"2024-01-21T03:21:13.379783+0000","last_active":"2024-01-21T03:56:20.721033+0000","last_peered":"2024-01-21T03:56:20.721033+0000","last_clean":"2024-01-21T03:56:20.721033+0000","last_became_active":"2024-01-21T03:21:13.373828+0000","last_became_peered":"2024-01-21T03:21:13.373828+0000","last_unstale":"2024-01-21T03:56:20.721033+0000","last_undegraded":"2024-01-21T03:56:20.721033+0000","last_fullsized":"2024-01-21T03:56:20.721033+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:54:16.898824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1229,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.720120+0000","last_change":"2024-01-21T03:21:11.007665+0000","last_active":"2024-01-21T03:56:19.720120+0000","last_peered":"2024-01-21T03:56:19.720120+0000","last_clean":"2024-01-21T03:56:19.720120+0000","last_became_active":"2024-01-21T03:21:10.999264+0000","last_became_peered":"2024-01-21T03:21:10.999264+0000","last_unstale":"2024-01-21T03:56:19.720120+0000","last_undegraded":"2024-01-21T03:56:19.720120+0000","last_fullsized":"2024-01-21T03:56:19.720120+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:43:27.507683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1226,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.245990+0000","last_change":"2024-01-21T03:21:15.387706+0000","last_active":"2024-01-21T03:56:20.245990+0000","last_peered":"2024-01-21T03:56:20.245990+0000","last_clean":"2024-01-21T03:56:20.245990+0000","last_became_active":"2024-01-21T03:21:15.068960+0000","last_became_peered":"2024-01-21T03:21:15.068960+0000","last_unstale":"2024-01-21T03:56:20.245990+0000","last_undegraded":"2024-01-21T03:56:20.245990+0000","last_fullsized":"2024-01-21T03:56:20.245990+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:01:23.405734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1232,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.696894+0000","last_change":"2024-01-21T03:21:33.019724+0000","last_active":"2024-01-21T03:56:19.696894+0000","last_peered":"2024-01-21T03:56:19.696894+0000","last_clean":"2024-01-21T03:56:19.696894+0000","last_became_active":"2024-01-21T03:21:32.997061+0000","last_became_peered":"2024-01-21T03:21:32.997061+0000","last_unstale":"2024-01-21T03:56:19.696894+0000","last_undegraded":"2024-01-21T03:56:19.696894+0000","last_fullsized":"2024-01-21T03:56:19.696894+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:52:53.090792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1218,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:20.266819+0000","last_change":"2024-01-21T03:21:14.774673+0000","last_active":"2024-01-21T03:56:20.266819+0000","last_peered":"2024-01-21T03:56:20.266819+0000","last_clean":"2024-01-21T03:56:20.266819+0000","last_became_active":"2024-01-21T03:21:14.761665+0000","last_became_peered":"2024-01-21T03:21:14.761665+0000","last_unstale":"2024-01-21T03:56:20.266819+0000","last_undegraded":"2024-01-21T03:56:20.266819+0000","last_fullsized":"2024-01-21T03:56:20.266819+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:09:47.334671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1232,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-01-21T03:56:19.715724+0000","last_change":"2024-01-21T03:21:34.181963+0000","last_active":"2024-01-21T03:56:19.715724+0000","last_peered":"2024-01-21T03:56:19.715724+0000","last_clean":"2024-01-21T03:56:19.715724+0000","last_became_active":"2024-01-21T03:21:34.168272+0000","last_became_peered":"2024-01-21T03:21:34.168272+0000","last_unstale":"2024-01-21T03:56:19.715724+0000","last_undegraded":"2024-01-21T03:56:19.715724+0000","last_fullsized":"2024-01-21T03:56:19.715724+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:16:18.395527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280948,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76408,"kb_used_data":4964,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295432,"statfs":{"total":96636764160,"available":96558522368,"internally_reserved":0,"allocated":5083136,"data_stored":2325045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":9.3070000000000004,"5min":22.673999999999999,"15min":23.361000000000001},"min":{"1min":5.9930000000000003,"5min":4.7569999999999997,"15min":4.2290000000000001},"max":{"1min":15.645,"5min":221.726,"15min":236.40199999999999},"last":8.2629999999999999},{"interface":"front","average":{"1min":12.923,"5min":28.297999999999998,"15min":22.507000000000001},"min":{"1min":7.3280000000000003,"5min":6.5869999999999997,"15min":4.423},"max":{"1min":21.279,"5min":315.84899999999999,"15min":317.16500000000002},"last":11.234}]},{"osd":1,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":12.68,"5min":24.010999999999999,"15min":26.009},"min":{"1min":7.8280000000000003,"5min":7.8280000000000003,"15min":6.7649999999999997},"max":{"1min":19.681999999999999,"5min":210.08799999999999,"15min":688.34500000000003},"last":11.881},{"interface":"front","average":{"1min":15.066000000000001,"5min":30.968,"15min":31.401},"min":{"1min":10.843999999999999,"5min":10.49,"15min":9.6310000000000002},"max":{"1min":23.169,"5min":288.46600000000001,"15min":686.94299999999998},"last":17.687999999999999}]},{"osd":2,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":9.9570000000000007,"5min":59.222999999999999,"15min":34.966999999999999},"min":{"1min":6.0730000000000004,"5min":6.0730000000000004,"15min":5.75},"max":{"1min":31.401,"5min":3500.2379999999998,"15min":3500.2379999999998},"last":8.6159999999999997},{"interface":"front","average":{"1min":14.031000000000001,"5min":37.201000000000001,"15min":29.867000000000001},"min":{"1min":7.9669999999999996,"5min":7.9669999999999996,"15min":7.4710000000000001},"max":{"1min":25.942,"5min":919.58199999999999,"15min":919.58199999999999},"last":13.473000000000001}]},{"osd":4,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":14.010999999999999,"5min":26.649000000000001,"15min":26.817},"min":{"1min":10.026,"5min":9.1489999999999991,"15min":8.9909999999999997},"max":{"1min":18.762,"5min":241.95599999999999,"15min":297.46600000000001},"last":12.843999999999999},{"interface":"front","average":{"1min":18.001999999999999,"5min":31.356999999999999,"15min":31.646999999999998},"min":{"1min":12.919,"5min":12.346,"15min":9.673},"max":{"1min":23.687000000000001,"5min":292.25400000000002,"15min":340.73500000000001},"last":19.527999999999999}]},{"osd":5,"last update":"Sun Jan 21 03:56:18 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":24.042999999999999,"15min":24.727},"min":{"1min":7.3680000000000003,"5min":7.3680000000000003,"15min":6.1989999999999998},"max":{"1min":22.300000000000001,"5min":214.58500000000001,"15min":362.22899999999998},"last":14.112},{"interface":"front","average":{"1min":13.831,"5min":26.408000000000001,"15min":27.535},"min":{"1min":8.9079999999999995,"5min":6.5439999999999996,"15min":6.5439999999999996},"max":{"1min":21.576000000000001,"5min":339.19499999999999,"15min":339.19499999999999},"last":16.254999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":18.814,"5min":31.257000000000001,"15min":30.303999999999998},"min":{"1min":11.645,"5min":11.206,"15min":7.1719999999999997},"max":{"1min":42.293999999999997,"5min":211.822,"15min":572.20899999999995},"last":16.989000000000001},{"interface":"front","average":{"1min":21.713000000000001,"5min":36.789999999999999,"15min":34.954999999999998},"min":{"1min":13.535,"5min":13.535,"15min":9.7240000000000002},"max":{"1min":45.457999999999998,"5min":215.315,"15min":376.66399999999999},"last":20.728000000000002}]},{"osd":7,"last update":"Sun Jan 21 03:56:18 2024","interfaces":[{"interface":"back","average":{"1min":15.071999999999999,"5min":23.681999999999999,"15min":23.119},"min":{"1min":9.3960000000000008,"5min":8.1329999999999991,"15min":5.9169999999999998},"max":{"1min":30.167999999999999,"5min":213.73500000000001,"15min":274.91000000000003},"last":10.551},{"interface":"front","average":{"1min":21.024000000000001,"5min":32.463999999999999,"15min":32.999000000000002},"min":{"1min":14.153,"5min":11.337,"15min":11.337},"max":{"1min":31.466000000000001,"5min":234.46299999999999,"15min":339.613},"last":21.956}]}]},{"osd":2,"up_from":21,"seq":90194313656,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68040,"kb_used_data":4924,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303800,"statfs":{"total":96636764160,"available":96567091200,"internally_reserved":0,"allocated":5042176,"data_stored":2303320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:55:51 2024","interfaces":[{"interface":"back","average":{"1min":11.087,"5min":20.286999999999999,"15min":17.802},"min":{"1min":4.9820000000000002,"5min":4.9820000000000002,"15min":4.5460000000000003},"max":{"1min":23.617000000000001,"5min":335.38200000000001,"15min":335.38200000000001},"last":5.5350000000000001},{"interface":"front","average":{"1min":9.5939999999999994,"5min":20.925999999999998,"15min":17.704000000000001},"min":{"1min":5.484,"5min":5.484,"15min":4.4580000000000002},"max":{"1min":22.068999999999999,"5min":343.05599999999998,"15min":343.05599999999998},"last":7.6950000000000003}]},{"osd":1,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":16.661999999999999,"5min":32.128,"15min":25.600999999999999},"min":{"1min":7.641,"5min":5.5960000000000001,"15min":5.2190000000000003},"max":{"1min":50.104999999999997,"5min":341.34399999999999,"15min":504.58100000000002},"last":9.5600000000000005},{"interface":"front","average":{"1min":14.928000000000001,"5min":35.445,"15min":27.994},"min":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":5.7350000000000003},"max":{"1min":32.649000000000001,"5min":674.13400000000001,"15min":674.13400000000001},"last":16.646000000000001}]},{"osd":3,"last update":"Sun Jan 21 03:55:37 2024","interfaces":[{"interface":"back","average":{"1min":14.738,"5min":22.449999999999999,"15min":20.140999999999998},"min":{"1min":7.9509999999999996,"5min":6.9820000000000002,"15min":6.3659999999999997},"max":{"1min":45.518999999999998,"5min":263.31400000000002,"15min":263.31400000000002},"last":12.964},{"interface":"front","average":{"1min":17.158999999999999,"5min":26.888000000000002,"15min":22.359999999999999},"min":{"1min":9.6389999999999993,"5min":8.1180000000000003,"15min":8.1180000000000003},"max":{"1min":50.167000000000002,"5min":683.75800000000004,"15min":683.75800000000004},"last":17.747}]},{"osd":4,"last update":"Sun Jan 21 03:55:26 2024","interfaces":[{"interface":"back","average":{"1min":17.934999999999999,"5min":27.861000000000001,"15min":24.173999999999999},"min":{"1min":9.5340000000000007,"5min":8.827,"15min":8.0609999999999999},"max":{"1min":34.628,"5min":687.15800000000002,"15min":687.15800000000002},"last":10.741},{"interface":"front","average":{"1min":19.096,"5min":24.669,"15min":22.831},"min":{"1min":11.587,"5min":8.8379999999999992,"15min":8.1620000000000008},"max":{"1min":41.676000000000002,"5min":318.77199999999999,"15min":318.77199999999999},"last":18.603000000000002}]},{"osd":5,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":19.896000000000001,"5min":30.462,"15min":24.710999999999999},"min":{"1min":7.9740000000000002,"5min":7.6660000000000004,"15min":6.0650000000000004},"max":{"1min":50.735999999999997,"5min":348.47000000000003,"15min":348.47000000000003},"last":14.537000000000001},{"interface":"front","average":{"1min":21.834,"5min":30.434999999999999,"15min":25.710000000000001},"min":{"1min":10.997999999999999,"5min":10.967000000000001,"15min":8.5210000000000008},"max":{"1min":52.564999999999998,"5min":264.41199999999998,"15min":264.41199999999998},"last":11.855}]},{"osd":6,"last update":"Sun Jan 21 03:55:26 2024","interfaces":[{"interface":"back","average":{"1min":19.821000000000002,"5min":31.231999999999999,"15min":27.202000000000002},"min":{"1min":12.441000000000001,"5min":8.5960000000000001,"15min":7.2800000000000002},"max":{"1min":52.945,"5min":679.51900000000001,"15min":679.51900000000001},"last":19.686},{"interface":"front","average":{"1min":21.942,"5min":34.5,"15min":28.763999999999999},"min":{"1min":14.582000000000001,"5min":10.416,"15min":10.121},"max":{"1min":53.688000000000002,"5min":676.11800000000005,"15min":676.11800000000005},"last":20.606999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":24.338999999999999,"5min":29.204999999999998,"15min":25.914000000000001},"min":{"1min":12.472,"5min":10.278,"15min":10.278},"max":{"1min":53.186999999999998,"5min":261.48899999999998,"15min":261.48899999999998},"last":13.686999999999999},{"interface":"front","average":{"1min":24.440000000000001,"5min":38.350999999999999,"15min":27.709},"min":{"1min":15.018000000000001,"5min":12.605,"15min":8.4239999999999995},"max":{"1min":42.475000000000001,"5min":678.09500000000003,"15min":678.09500000000003},"last":21.939}]}]},{"osd":1,"up_from":21,"seq":90194313655,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94744,"kb_used_data":5316,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94277096,"statfs":{"total":96636764160,"available":96539746304,"internally_reserved":0,"allocated":5443584,"data_stored":2701160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":11.568,"5min":16.558,"15min":21.972999999999999},"min":{"1min":4.5970000000000004,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":26.126000000000001,"5min":154.52799999999999,"15min":528.255},"last":6.4720000000000004},{"interface":"front","average":{"1min":12.198,"5min":15.057,"15min":19.050999999999998},"min":{"1min":4.9619999999999997,"5min":4.5199999999999996,"15min":4.3079999999999998},"max":{"1min":24.762,"5min":127.73999999999999,"15min":447.83199999999999},"last":5.5960000000000001}]},{"osd":2,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":13.651,"5min":17.736999999999998,"15min":23.350999999999999},"min":{"1min":8.5549999999999997,"5min":7.8719999999999999,"15min":6.194},"max":{"1min":21.834,"5min":116.48999999999999,"15min":420.87700000000001},"last":11.682},{"interface":"front","average":{"1min":15.859999999999999,"5min":26.047999999999998,"15min":27.696999999999999},"min":{"1min":8.5239999999999991,"5min":7.9889999999999999,"15min":7.3209999999999997},"max":{"1min":30.548999999999999,"5min":265.24200000000002,"15min":451.637},"last":12.586}]},{"osd":3,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":20.800000000000001,"5min":22.111000000000001,"15min":23.620999999999999},"min":{"1min":9.2140000000000004,"5min":8.0370000000000008,"15min":8.0370000000000008},"max":{"1min":62.801000000000002,"5min":141.52099999999999,"15min":419.892},"last":13.31},{"interface":"front","average":{"1min":28.942,"5min":25.436,"15min":27.707000000000001},"min":{"1min":8.9109999999999996,"5min":8.6479999999999997,"15min":8.3059999999999992},"max":{"1min":123.798,"5min":143.58699999999999,"15min":425.76100000000002},"last":18.611999999999998}]},{"osd":4,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":26.920999999999999,"5min":26.507999999999999,"15min":25.122},"min":{"1min":11.009,"5min":7.4210000000000003,"15min":7.4210000000000003},"max":{"1min":106.116,"5min":229.63200000000001,"15min":344.48200000000003},"last":13.99},{"interface":"front","average":{"1min":25.748999999999999,"5min":29.904,"15min":30.959},"min":{"1min":12.282,"5min":8.3059999999999992,"15min":8.1259999999999994},"max":{"1min":65.070999999999998,"5min":236.571,"15min":385.43000000000001},"last":19.704999999999998}]},{"osd":5,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":22.766999999999999,"5min":28.076000000000001,"15min":30.946000000000002},"min":{"1min":13.015000000000001,"5min":9.5839999999999996,"15min":9.5839999999999996},"max":{"1min":38.545999999999999,"5min":138.84899999999999,"15min":338.88200000000001},"last":20.771000000000001},{"interface":"front","average":{"1min":18.858000000000001,"5min":26.266999999999999,"15min":29.963999999999999},"min":{"1min":7.577,"5min":7.577,"15min":7.577},"max":{"1min":34.542999999999999,"5min":187.03,"15min":356.423},"last":14.676}]},{"osd":6,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":21.745000000000001,"5min":31.524000000000001,"15min":31.725000000000001},"min":{"1min":10.369999999999999,"5min":10.369999999999999,"15min":9.7289999999999992},"max":{"1min":53.404000000000003,"5min":280.85000000000002,"15min":409.08100000000002},"last":15.343999999999999},{"interface":"front","average":{"1min":20.478000000000002,"5min":25.875,"15min":23.920000000000002},"min":{"1min":6.6100000000000003,"5min":6.6100000000000003,"15min":6.6100000000000003},"max":{"1min":59.128999999999998,"5min":243.083,"15min":261.75099999999998},"last":9.7629999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":21.91,"5min":28.564,"15min":25.913},"min":{"1min":12.441000000000001,"5min":11.923,"15min":7.21},"max":{"1min":43.122999999999998,"5min":169.571,"15min":357.57299999999998},"last":16.021000000000001},{"interface":"front","average":{"1min":23.428000000000001,"5min":23.902999999999999,"15min":25.715},"min":{"1min":8.6929999999999996,"5min":8.1959999999999997,"15min":7.9729999999999999},"max":{"1min":63.999000000000002,"5min":176.809,"15min":418.70600000000002},"last":10.644}]}]},{"osd":0,"up_from":21,"seq":90194313659,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72244,"kb_used_data":4964,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299596,"statfs":{"total":96636764160,"available":96562786304,"internally_reserved":0,"allocated":5083136,"data_stored":2325045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":15.691000000000001,"5min":26.472999999999999,"15min":22.228000000000002},"min":{"1min":6.7560000000000002,"5min":6.0220000000000002,"15min":5.7869999999999999},"max":{"1min":49.520000000000003,"5min":246.35400000000001,"15min":301.80599999999998},"last":8.3079999999999998},{"interface":"front","average":{"1min":21.231000000000002,"5min":29.074000000000002,"15min":24.678000000000001},"min":{"1min":11.365,"5min":6.0270000000000001,"15min":6.0270000000000001},"max":{"1min":51.290999999999997,"5min":232.851,"15min":242.80000000000001},"last":21.334}]},{"osd":2,"last update":"Sun Jan 21 03:55:44 2024","interfaces":[{"interface":"back","average":{"1min":16.657,"5min":32.222999999999999,"15min":27.050999999999998},"min":{"1min":9.2360000000000007,"5min":7.2160000000000002,"15min":6.6289999999999996},"max":{"1min":28.940000000000001,"5min":279.71300000000002,"15min":312.15800000000002},"last":18.870000000000001},{"interface":"front","average":{"1min":12.459,"5min":35.030999999999999,"15min":27.09},"min":{"1min":6.9169999999999998,"5min":6.3959999999999999,"15min":5.0910000000000002},"max":{"1min":23.015000000000001,"5min":332.57499999999999,"15min":357.38900000000001},"last":17.791}]},{"osd":3,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":22.167999999999999,"5min":43.600000000000001,"15min":32.090000000000003},"min":{"1min":7.5049999999999999,"5min":7.5049999999999999,"15min":6.9829999999999997},"max":{"1min":68.963999999999999,"5min":547.15800000000002,"15min":547.15800000000002},"last":20.677},{"interface":"front","average":{"1min":20.045999999999999,"5min":44.688000000000002,"15min":32.734999999999999},"min":{"1min":10.461,"5min":7.5979999999999999,"15min":7.5979999999999999},"max":{"1min":74.480999999999995,"5min":361.11500000000001,"15min":361.11500000000001},"last":26.079999999999998}]},{"osd":4,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":22.556000000000001,"5min":35.131,"15min":25.462},"min":{"1min":10.24,"5min":7.4630000000000001,"15min":7.3019999999999996},"max":{"1min":75.359999999999999,"5min":333.23899999999998,"15min":333.23899999999998},"last":27.132999999999999},{"interface":"front","average":{"1min":27.826000000000001,"5min":41.021000000000001,"15min":28.832999999999998},"min":{"1min":14.176,"5min":10.836,"15min":6.3630000000000004},"max":{"1min":90.658000000000001,"5min":333.87400000000002,"15min":333.87400000000002},"last":29.652000000000001}]},{"osd":5,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":12.843,"5min":40.453000000000003,"15min":31.494},"min":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"max":{"1min":66.775999999999996,"5min":387.58100000000002,"15min":543.84100000000001},"last":6.617},{"interface":"front","average":{"1min":26.053999999999998,"5min":30.795999999999999,"15min":23.568000000000001},"min":{"1min":10.856999999999999,"5min":6.3529999999999998,"15min":5.452},"max":{"1min":88.313999999999993,"5min":266.03800000000001,"15min":335.589},"last":23.265000000000001}]},{"osd":6,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":16.995999999999999,"5min":31.850000000000001,"15min":26.893999999999998},"min":{"1min":9.0709999999999997,"5min":6.1059999999999999,"15min":6.1059999999999999},"max":{"1min":67.790999999999997,"5min":311.358,"15min":370.30200000000002},"last":30.492000000000001},{"interface":"front","average":{"1min":25.103000000000002,"5min":33.008000000000003,"15min":26.288},"min":{"1min":12.054,"5min":8.2769999999999992,"15min":7.2389999999999999},"max":{"1min":70.632000000000005,"5min":230.27699999999999,"15min":230.27699999999999},"last":19.509}]},{"osd":7,"last update":"Sun Jan 21 03:55:28 2024","interfaces":[{"interface":"back","average":{"1min":24.888000000000002,"5min":30.061,"15min":25.626000000000001},"min":{"1min":10.714,"5min":9.1110000000000007,"15min":8.8960000000000008},"max":{"1min":71.314999999999998,"5min":131.65000000000001,"15min":214.071},"last":16.683},{"interface":"front","average":{"1min":28.067,"5min":38.735999999999997,"15min":28.626000000000001},"min":{"1min":12.391,"5min":8.8070000000000004,"15min":8.8070000000000004},"max":{"1min":94.670000000000002,"5min":348.38600000000002,"15min":348.38600000000002},"last":25.440999999999999}]}]},{"osd":6,"up_from":18,"seq":77309411770,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72208,"kb_used_data":4928,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299632,"statfs":{"total":96636764160,"available":96562823168,"internally_reserved":0,"allocated":5046272,"data_stored":2303325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":16.803000000000001,"5min":22.076000000000001,"15min":21.289999999999999},"min":{"1min":5.3380000000000001,"5min":5.3380000000000001,"15min":4.391},"max":{"1min":98.180000000000007,"5min":284.42500000000001,"15min":438.98899999999998},"last":12.811},{"interface":"front","average":{"1min":28.359999999999999,"5min":27.478999999999999,"15min":21.431000000000001},"min":{"1min":8.7149999999999999,"5min":5.2160000000000002,"15min":4.8840000000000003},"max":{"1min":144.92400000000001,"5min":380.07600000000002,"15min":380.07600000000002},"last":10.606999999999999}]},{"osd":1,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":19.898,"5min":31.847999999999999,"15min":24.283000000000001},"min":{"1min":8.7289999999999992,"5min":6.883,"15min":4.7389999999999999},"max":{"1min":54.722999999999999,"5min":526.48099999999999,"15min":526.48099999999999},"last":7.7670000000000003},{"interface":"front","average":{"1min":14.859999999999999,"5min":26.492000000000001,"15min":22.867999999999999},"min":{"1min":6.3600000000000003,"5min":4.3120000000000003,"15min":4.3120000000000003},"max":{"1min":44.970999999999997,"5min":395.72399999999999,"15min":395.72399999999999},"last":9.3170000000000002}]},{"osd":2,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":12.648,"5min":21.052,"15min":20.300999999999998},"min":{"1min":6.0350000000000001,"5min":5.8920000000000003,"15min":5.8920000000000003},"max":{"1min":35.268999999999998,"5min":209.78,"15min":213.83199999999999},"last":12.169},{"interface":"front","average":{"1min":22.007000000000001,"5min":31.407,"15min":26.654},"min":{"1min":6.7210000000000001,"5min":6.7210000000000001,"15min":6.7210000000000001},"max":{"1min":77.159000000000006,"5min":444.822,"15min":444.822},"last":26.718}]},{"osd":3,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":18.542999999999999,"5min":33.561,"15min":26.021000000000001},"min":{"1min":8.1959999999999997,"5min":6.1890000000000001,"15min":6.1890000000000001},"max":{"1min":59.645000000000003,"5min":527.40899999999999,"15min":527.40899999999999},"last":27.533000000000001},{"interface":"front","average":{"1min":17.805,"5min":25.48,"15min":23.213999999999999},"min":{"1min":6.4089999999999998,"5min":6.4050000000000002,"15min":6.4050000000000002},"max":{"1min":57.729999999999997,"5min":370.10000000000002,"15min":370.10000000000002},"last":14.664999999999999}]},{"osd":4,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":21.780000000000001,"5min":31.358000000000001,"15min":24.521999999999998},"min":{"1min":6.9160000000000004,"5min":6.3120000000000003,"15min":6.3120000000000003},"max":{"1min":95.962000000000003,"5min":385.49200000000002,"15min":385.49200000000002},"last":6.5410000000000004},{"interface":"front","average":{"1min":21.581,"5min":26.248000000000001,"15min":21.954000000000001},"min":{"1min":8.0660000000000007,"5min":7.2130000000000001,"15min":6.7539999999999996},"max":{"1min":89.241,"5min":241.59200000000001,"15min":241.59200000000001},"last":9.9629999999999992}]},{"osd":5,"last update":"Sun Jan 21 03:55:30 2024","interfaces":[{"interface":"back","average":{"1min":20.192,"5min":27.117000000000001,"15min":25.914000000000001},"min":{"1min":8.4700000000000006,"5min":8.2810000000000006,"15min":7.5679999999999996},"max":{"1min":63.576999999999998,"5min":488.01299999999998,"15min":488.01299999999998},"last":25.760000000000002},{"interface":"front","average":{"1min":18.07,"5min":21.914999999999999,"15min":26.193999999999999},"min":{"1min":9.5719999999999992,"5min":8.1890000000000001,"15min":8.1890000000000001},"max":{"1min":32.356000000000002,"5min":197.38,"15min":333.43900000000002},"last":13.863}]},{"osd":7,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":23.504000000000001,"5min":30.591999999999999,"15min":25.545000000000002},"min":{"1min":10.676,"5min":7.1340000000000003,"15min":7.1340000000000003},"max":{"1min":100.84399999999999,"5min":322.54599999999999,"15min":322.54599999999999},"last":8.3339999999999996},{"interface":"front","average":{"1min":24.988,"5min":29.010999999999999,"15min":25.050000000000001},"min":{"1min":13.025,"5min":9.6940000000000008,"15min":8.6699999999999999},"max":{"1min":90.594999999999999,"5min":218.30199999999999,"15min":218.30199999999999},"last":15.365}]}]},{"osd":5,"up_from":18,"seq":77309411773,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72600,"kb_used_data":5316,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299240,"statfs":{"total":96636764160,"available":96562421760,"internally_reserved":0,"allocated":5443584,"data_stored":2701160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":9.1660000000000004,"5min":22.483000000000001,"15min":20.535},"min":{"1min":4.9420000000000002,"5min":4.6539999999999999,"15min":4.0940000000000003},"max":{"1min":28.151,"5min":241.49000000000001,"15min":346.541},"last":7.2160000000000002},{"interface":"front","average":{"1min":13.221,"5min":26.934000000000001,"15min":26.605},"min":{"1min":7.6550000000000002,"5min":6.1500000000000004,"15min":5.8819999999999997},"max":{"1min":24.053000000000001,"5min":236.81100000000001,"15min":524.904},"last":12.234999999999999}]},{"osd":1,"last update":"Sun Jan 21 03:55:22 2024","interfaces":[{"interface":"back","average":{"1min":14.785,"5min":26.623999999999999,"15min":20.832999999999998},"min":{"1min":6.2729999999999997,"5min":5.1429999999999998,"15min":4.907},"max":{"1min":84.840999999999994,"5min":284.07400000000001,"15min":368.12400000000002},"last":8.4740000000000002},{"interface":"front","average":{"1min":20.699999999999999,"5min":34.825000000000003,"15min":27.59},"min":{"1min":11.284000000000001,"5min":8.8569999999999993,"15min":7.5359999999999996},"max":{"1min":85.671000000000006,"5min":296.56400000000002,"15min":368.86099999999999},"last":10.801}]},{"osd":2,"last update":"Sun Jan 21 03:55:35 2024","interfaces":[{"interface":"back","average":{"1min":11.273999999999999,"5min":46.090000000000003,"15min":29.077000000000002},"min":{"1min":7.25,"5min":6.6310000000000002,"15min":6.0579999999999998},"max":{"1min":27.489000000000001,"5min":2166.2739999999999,"15min":2166.2739999999999},"last":10.039999999999999},{"interface":"front","average":{"1min":19.923999999999999,"5min":53.329000000000001,"15min":37.494},"min":{"1min":9.8030000000000008,"5min":7.1150000000000002,"15min":7.1150000000000002},"max":{"1min":71.177999999999997,"5min":2181.6190000000001,"15min":2181.6190000000001},"last":15.766}]},{"osd":3,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":16.638000000000002,"5min":30.282,"15min":26.837},"min":{"1min":6.8179999999999996,"5min":6.2770000000000001,"15min":6.2770000000000001},"max":{"1min":63.612000000000002,"5min":412.714,"15min":412.714},"last":16.672999999999998},{"interface":"front","average":{"1min":20.707999999999998,"5min":34.683999999999997,"15min":28.085999999999999},"min":{"1min":9.9399999999999995,"5min":9.6389999999999993,"15min":6.8840000000000003},"max":{"1min":57.048000000000002,"5min":314.28199999999998,"15min":314.28199999999998},"last":13.682}]},{"osd":4,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":21.478000000000002,"5min":39.764000000000003,"15min":27.739999999999998},"min":{"1min":12.226000000000001,"5min":7.7060000000000004,"15min":6.274},"max":{"1min":49.774999999999999,"5min":418.18299999999999,"15min":418.18299999999999},"last":18.27},{"interface":"front","average":{"1min":24.219000000000001,"5min":42.651000000000003,"15min":30.201000000000001},"min":{"1min":14.734999999999999,"5min":11.993,"15min":7.5490000000000004},"max":{"1min":57.712000000000003,"5min":387.125,"15min":387.125},"last":19.117999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":17.492999999999999,"5min":32.762,"15min":25.588999999999999},"min":{"1min":8.6739999999999995,"5min":6.7050000000000001,"15min":6.7050000000000001},"max":{"1min":31.645,"5min":431.40899999999999,"15min":431.40899999999999},"last":11.577},{"interface":"front","average":{"1min":23.259,"5min":36.051000000000002,"15min":28.965},"min":{"1min":11.241,"5min":9.0809999999999995,"15min":8.5470000000000006},"max":{"1min":61.225999999999999,"5min":432.19,"15min":432.19},"last":13.022}]},{"osd":7,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":23.452000000000002,"5min":38.652000000000001,"15min":29.713000000000001},"min":{"1min":13.459,"5min":9.7560000000000002,"15min":7.8639999999999999},"max":{"1min":48.789999999999999,"5min":281.06099999999998,"15min":281.06099999999998},"last":17.498999999999999},{"interface":"front","average":{"1min":12.581,"5min":26.823,"15min":22.974},"min":{"1min":6.5170000000000003,"5min":6.5170000000000003,"15min":6.5170000000000003},"max":{"1min":36.563000000000002,"5min":282.29000000000002,"15min":282.29000000000002},"last":9.2140000000000004}]}]},{"osd":7,"up_from":14,"seq":60129542586,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72224,"kb_used_data":4944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299616,"statfs":{"total":96636764160,"available":96562806784,"internally_reserved":0,"allocated":5062656,"data_stored":2314831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:17 2024","interfaces":[{"interface":"back","average":{"1min":11.395,"5min":20.713000000000001,"15min":16.225999999999999},"min":{"1min":5.0099999999999998,"5min":4.7450000000000001,"15min":4.1929999999999996},"max":{"1min":27.079000000000001,"5min":239.37799999999999,"15min":304.20800000000003},"last":26.015999999999998},{"interface":"front","average":{"1min":10.247999999999999,"5min":21.539000000000001,"15min":16.581},"min":{"1min":5.9039999999999999,"5min":5.2709999999999999,"15min":4.2679999999999998},"max":{"1min":31.059999999999999,"5min":181.04599999999999,"15min":181.04599999999999},"last":22.859000000000002}]},{"osd":1,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":12.342000000000001,"5min":19.533000000000001,"15min":20.614000000000001},"min":{"1min":6.0869999999999997,"5min":5.6580000000000004,"15min":5.0419999999999998},"max":{"1min":27.402999999999999,"5min":138.82499999999999,"15min":248.17500000000001},"last":13.691000000000001},{"interface":"front","average":{"1min":14.029,"5min":23.824999999999999,"15min":23.670999999999999},"min":{"1min":7.9109999999999996,"5min":6.3769999999999998,"15min":6.3769999999999998},"max":{"1min":26.454000000000001,"5min":214.55000000000001,"15min":277.85700000000003},"last":11.555}]},{"osd":2,"last update":"Sun Jan 21 03:56:17 2024","interfaces":[{"interface":"back","average":{"1min":17.594999999999999,"5min":36.784999999999997,"15min":27.683},"min":{"1min":7.2869999999999999,"5min":7.2290000000000001,"15min":6.1779999999999999},"max":{"1min":47.972999999999999,"5min":683.21000000000004,"15min":683.21000000000004},"last":18.545000000000002},{"interface":"front","average":{"1min":16.43,"5min":24.434999999999999,"15min":22.135999999999999},"min":{"1min":9.0399999999999991,"5min":6.3890000000000002,"15min":6.3890000000000002},"max":{"1min":35.869,"5min":150.369,"15min":367.30000000000001},"last":15.68}]},{"osd":3,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":17.163,"5min":27.100000000000001,"15min":22.632000000000001},"min":{"1min":10.734999999999999,"5min":7.1470000000000002,"15min":6.8040000000000003},"max":{"1min":24.895,"5min":192.14599999999999,"15min":192.14599999999999},"last":12.595000000000001},{"interface":"front","average":{"1min":16.158000000000001,"5min":24.597000000000001,"15min":21.649000000000001},"min":{"1min":7.851,"5min":7.8090000000000002,"15min":6.5960000000000001},"max":{"1min":29.548999999999999,"5min":166.846,"15min":210.006},"last":9.8450000000000006}]},{"osd":4,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":20.638999999999999,"5min":27.516999999999999,"15min":22.658999999999999},"min":{"1min":9.4700000000000006,"5min":7.6260000000000003,"15min":7.2309999999999999},"max":{"1min":43.503,"5min":197.31,"15min":268.416},"last":30.398},{"interface":"front","average":{"1min":17.760999999999999,"5min":25.548999999999999,"15min":21.73},"min":{"1min":7.516,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":33.429000000000002,"5min":143.07400000000001,"15min":168.95599999999999},"last":28.981999999999999}]},{"osd":5,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":17.228999999999999,"5min":25.649999999999999,"15min":24.148},"min":{"1min":7.3029999999999999,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":25.670000000000002,"5min":149.56,"15min":226.947},"last":20.795999999999999},{"interface":"front","average":{"1min":17.081,"5min":24.050000000000001,"15min":23.798999999999999},"min":{"1min":8.6419999999999995,"5min":7.5839999999999996,"15min":7.5839999999999996},"max":{"1min":41.097000000000001,"5min":137.00200000000001,"15min":248.94800000000001},"last":28.298999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":21.542000000000002,"5min":27.488,"15min":26.065999999999999},"min":{"1min":10.897,"5min":9.4459999999999997,"15min":9.4459999999999997},"max":{"1min":55.081000000000003,"5min":193.16,"15min":295.12099999999998},"last":27.280000000000001},{"interface":"front","average":{"1min":19.73,"5min":28.16,"15min":26.513999999999999},"min":{"1min":11.029,"5min":9.3219999999999992,"15min":9.3219999999999992},"max":{"1min":55.758000000000003,"5min":178.01900000000001,"15min":278.71899999999999},"last":23.907}]}]},{"osd":4,"up_from":14,"seq":60129542585,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72220,"kb_used_data":4936,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299620,"statfs":{"total":96636764160,"available":96562810880,"internally_reserved":0,"allocated":5054464,"data_stored":2310730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":8.7889999999999997,"5min":15.209,"15min":16.693000000000001},"min":{"1min":4.9939999999999998,"5min":4.4660000000000002,"15min":4.4100000000000001},"max":{"1min":18.459,"5min":220.72900000000001,"15min":220.72900000000001},"last":5.0410000000000004},{"interface":"front","average":{"1min":12.965,"5min":16.300000000000001,"15min":18.149999999999999},"min":{"1min":6.6360000000000001,"5min":4.6029999999999998,"15min":4.6029999999999998},"max":{"1min":45.616,"5min":101.768,"15min":204.142},"last":23.486999999999998}]},{"osd":1,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":12.475,"5min":22.411999999999999,"15min":18.585000000000001},"min":{"1min":6.6440000000000001,"5min":4.9359999999999999,"15min":4.9359999999999999},"max":{"1min":26.292999999999999,"5min":272.61399999999998,"15min":272.61399999999998},"last":20.783999999999999},{"interface":"front","average":{"1min":16.969000000000001,"5min":24.193000000000001,"15min":20.829000000000001},"min":{"1min":9.1150000000000002,"5min":6.8719999999999999,"15min":5.6200000000000001},"max":{"1min":49.414999999999999,"5min":142.97300000000001,"15min":223.49299999999999},"last":34.969000000000001}]},{"osd":2,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":23.027000000000001,"5min":29.632000000000001,"15min":24.187999999999999},"min":{"1min":9.3979999999999997,"5min":7.4320000000000004,"15min":6.6879999999999997},"max":{"1min":77.858999999999995,"5min":285.61000000000001,"15min":285.61000000000001},"last":25.93},{"interface":"front","average":{"1min":21.07,"5min":26.555,"15min":22.710999999999999},"min":{"1min":7.6769999999999996,"5min":6.899,"15min":6.2080000000000002},"max":{"1min":101.676,"5min":298.892,"15min":298.892},"last":8.9760000000000009}]},{"osd":3,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":14.119,"5min":19.032,"15min":20.623999999999999},"min":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"max":{"1min":48.497,"5min":132.958,"15min":200.697},"last":29.013999999999999},{"interface":"front","average":{"1min":19.265000000000001,"5min":23.931999999999999,"15min":21.149000000000001},"min":{"1min":8.9890000000000008,"5min":7.5250000000000004,"15min":7.1539999999999999},"max":{"1min":60.786000000000001,"5min":166.66200000000001,"15min":188.06399999999999},"last":24.388000000000002}]},{"osd":5,"last update":"Sun Jan 21 03:55:58 2024","interfaces":[{"interface":"back","average":{"1min":17.315000000000001,"5min":26.736000000000001,"15min":24.969000000000001},"min":{"1min":10.885999999999999,"5min":8.2970000000000006,"15min":6.7359999999999998},"max":{"1min":30.718,"5min":182.53700000000001,"15min":470.87700000000001},"last":19.791},{"interface":"front","average":{"1min":13.831,"5min":23.012,"15min":24.087},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":6.944},"max":{"1min":23.405000000000001,"5min":145.125,"15min":471.56999999999999},"last":26.879999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:55:37 2024","interfaces":[{"interface":"back","average":{"1min":21.370999999999999,"5min":25.059999999999999,"15min":21.741},"min":{"1min":14.724,"5min":8.5820000000000007,"15min":6.415},"max":{"1min":73.459999999999994,"5min":208.42099999999999,"15min":285.87},"last":30.898},{"interface":"front","average":{"1min":18.635000000000002,"5min":26.960000000000001,"15min":24.614000000000001},"min":{"1min":9.5129999999999999,"5min":9.5129999999999999,"15min":7.5430000000000001},"max":{"1min":64.510000000000005,"5min":252.47999999999999,"15min":252.47999999999999},"last":25.187999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:55:58 2024","interfaces":[{"interface":"back","average":{"1min":20.036999999999999,"5min":25.931000000000001,"15min":22.280000000000001},"min":{"1min":12.244,"5min":9.3040000000000003,"15min":7.1820000000000004},"max":{"1min":44.164999999999999,"5min":141.51300000000001,"15min":229.04900000000001},"last":28.405000000000001},{"interface":"front","average":{"1min":22.382999999999999,"5min":28.206,"15min":22.698},"min":{"1min":15.759,"5min":12.417999999999999,"15min":7.4299999999999997},"max":{"1min":42.031999999999996,"5min":179.12799999999999,"15min":179.12799999999999},"last":31.728000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T03:56:27.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-21T03:56:27.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.672+0000 7f90662d7640 10 monclient: get_monmap_and_config 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.672+0000 7f90662d7640 10 monclient: build_initial_monmap 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.672+0000 7f90662d7640 10 monclient: monmap: 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:27.673360+0000 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:27.673360+0000 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:27.673 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:27.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 1 Processor -- start 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 1 -- start start 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 10 monclient: init 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f90600a8a40 0x7f9060145b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 10 monclient(hunting): picked mon.noname-c con 0x7f90600a8a40 addr v2:172.21.15.177:3300/0 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f90601460a0 0x7f906013f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 10 monclient(hunting): picked mon.noname-a con 0x7f90601460a0 addr v2:172.21.15.7:3300/0 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 0x7f906013f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 10 monclient(hunting): picked mon.noname-b con 0x7f90600b21c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90600aa7d0 con 0x7f90601460a0 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90600aa9b0 con 0x7f90600a8a40 2024-01-21T03:56:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90600aac90 con 0x7f90600b21c0 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.673+0000 7f90662d7640 10 monclient(hunting): authenticate will time out at 3533.989576s 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.674+0000 7f9064874640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 0x7f906013f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.674+0000 7f905ffff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f90600a8a40 0x7f9060145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.674+0000 7f9064874640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 0x7f906013f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:36978/0 (socket says 172.21.15.7:36978) 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.674+0000 7f9064874640 1 -- 172.21.15.7:0/1470427395 learned_addr learned my addr 172.21.15.7:0/1470427395 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.674+0000 7f9064874640 10 monclient(hunting): get_auth_request con 0x7f90600b21c0 auth_method 0 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.674+0000 7f9064874640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.674+0000 7f9064874640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.674+0000 7f9064874640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.675+0000 7f905ffff640 10 monclient(hunting): get_auth_request con 0x7f90600a8a40 auth_method 0 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.675+0000 7f905ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.675+0000 7f905ffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.675+0000 7f905ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.675+0000 7f9064874640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:27.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.675+0000 7f9064874640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:27.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.675+0000 7f9064874640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:27.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.675+0000 7f905ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:27.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.675+0000 7f905ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:27.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.675+0000 7f905ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:27.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.676+0000 7f905f7fe640 1 --2- 172.21.15.7:0/1470427395 >> v2:172.21.15.7:3300/0 conn(0x7f90601460a0 0x7f906013f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:27.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.677+0000 7f9064874640 10 monclient(hunting): handle_auth_done global_id 17996 payload 274 2024-01-21T03:56:27.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.677+0000 7f9064874640 1 -- 172.21.15.7:0/1470427395 >> v2:172.21.15.177:3300/0 conn(0x7f90600a8a40 msgr2=0x7f9060145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:27.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.677+0000 7f9064874640 1 --2- 172.21.15.7:0/1470427395 >> v2:172.21.15.177:3300/0 conn(0x7f90600a8a40 0x7f9060145b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.677+0000 7f9064874640 1 -- 172.21.15.7:0/1470427395 >> v2:172.21.15.7:3300/0 conn(0x7f90601460a0 msgr2=0x7f906013f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:27.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.677+0000 7f9064874640 1 --2- 172.21.15.7:0/1470427395 >> v2:172.21.15.7:3300/0 conn(0x7f90601460a0 0x7f906013f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.677+0000 7f9064874640 10 monclient: _finish_hunting 0 2024-01-21T03:56:27.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.677+0000 7f9064874640 1 monclient: found mon.noname-b 2024-01-21T03:56:27.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.677+0000 7f9064874640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:27.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.677+0000 7f9064874640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:56:27.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.677+0000 7f9064874640 1 -- 172.21.15.7:0/1470427395 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90600aaf70 con 0x7f90600b21c0 2024-01-21T03:56:27.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.679+0000 7f9064874640 1 --2- 172.21.15.7:0/1470427395 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 0x7f906013f6e0 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 crypto rx=0x7f90600b5090 tx=0x7f905802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35d94727c038c419 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:27.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f905effd640 1 -- 172.21.15.7:0/1470427395 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905802cb00 con 0x7f90600b21c0 2024-01-21T03:56:27.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f905effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:27.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f905effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:27.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f905effd640 10 monclient: dump: 2024-01-21T03:56:27.684 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:27.684 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:27.684 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.684 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.684 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f905effd640 10 monclient: _finish_auth 0 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f905effd640 10 monclient: _check_auth_tickets 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f905effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f905effd640 1 -- 172.21.15.7:0/1470427395 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905802cc60 con 0x7f90600b21c0 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f90662d7640 5 monclient: authenticate success, global_id 17996 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f90662d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f905effd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f90662d7640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:27.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f905effd640 1 -- 172.21.15.7:0/1470427395 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90580343f0 con 0x7f90600b21c0 2024-01-21T03:56:27.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f905effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:27.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f90662d7640 4 set_mon_vals no callback set 2024-01-21T03:56:27.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f905effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:27.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.682+0000 7f905effd640 10 monclient: dump: 2024-01-21T03:56:27.686 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:27.686 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:27.686 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.686 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.686 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:27.686 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:27.686 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:27.686 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:27.687 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:27.687 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:27.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.683+0000 7f90662d7640 10 monclient: shutdown 2024-01-21T03:56:27.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.683+0000 7f90662d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:27.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.683+0000 7f90662d7640 1 -- 172.21.15.7:0/1470427395 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 msgr2=0x7f906013f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:27.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.683+0000 7f90662d7640 1 --2- 172.21.15.7:0/1470427395 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 0x7f906013f6e0 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 crypto rx=0x7f90600b5090 tx=0x7f905802bf60 comp rx=0 tx=0).stop 2024-01-21T03:56:27.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.683+0000 7f90662d7640 1 -- 172.21.15.7:0/1470427395 shutdown_connections 2024-01-21T03:56:27.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.683+0000 7f90662d7640 1 --2- 172.21.15.7:0/1470427395 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 0x7f906013f6e0 unknown :-1 s=CLOSED pgs=3148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.683+0000 7f90662d7640 1 --2- 172.21.15.7:0/1470427395 >> v2:172.21.15.7:3300/0 conn(0x7f90601460a0 0x7f906013f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.683+0000 7f90662d7640 1 --2- 172.21.15.7:0/1470427395 >> v2:172.21.15.177:3300/0 conn(0x7f90600a8a40 0x7f9060145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.683+0000 7f90662d7640 1 -- 172.21.15.7:0/1470427395 >> 172.21.15.7:0/1470427395 conn(0x7f9060066f50 msgr2=0x7f90600b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:27.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.683+0000 7f90662d7640 1 -- 172.21.15.7:0/1470427395 shutdown_connections 2024-01-21T03:56:27.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.683+0000 7f90662d7640 1 -- 172.21.15.7:0/1470427395 wait complete. 2024-01-21T03:56:27.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.684+0000 7f90662d7640 10 monclient: build_initial_monmap 2024-01-21T03:56:27.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.684+0000 7f90662d7640 10 monclient: monmap: 2024-01-21T03:56:27.690 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:27.690 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:27.690 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:27.690 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:27.690 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:27.691 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.684+0000 7f90662d7640 1 librados: starting msgr at 2024-01-21T03:56:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.684+0000 7f90662d7640 1 librados: starting objecter 2024-01-21T03:56:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.684+0000 7f90662d7640 1 Processor -- start 2024-01-21T03:56:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.684+0000 7f90662d7640 1 -- start start 2024-01-21T03:56:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.684+0000 7f90662d7640 1 librados: setting wanted keys 2024-01-21T03:56:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.684+0000 7f90662d7640 1 librados: calling monclient init 2024-01-21T03:56:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.684+0000 7f90662d7640 10 monclient: init 2024-01-21T03:56:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90600a8a40 0x7f9060144e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 10 monclient(hunting): picked mon.noname-b con 0x7f90600a8a40 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 0x7f90600b7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 10 monclient(hunting): picked mon.noname-c con 0x7f90600b21c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90601460a0 0x7f90600b8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 10 monclient(hunting): picked mon.noname-a con 0x7f90601460a0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90600b61a0 con 0x7f90601460a0 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9060141940 con 0x7f90600a8a40 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90600b85d0 con 0x7f90600b21c0 2024-01-21T03:56:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f90662d7640 10 monclient(hunting): authenticate will time out at 3534.001215s 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f905f7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 0x7f90600b7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.685+0000 7f905ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90600a8a40 0x7f9060144e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905f7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 0x7f90600b7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:36988/0 (socket says 172.21.15.7:36988) 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905f7fe640 1 -- 172.21.15.7:0/2112668759 learned_addr learned my addr 172.21.15.7:0/2112668759 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905f7fe640 10 monclient(hunting): get_auth_request con 0x7f90600b21c0 auth_method 0 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905ffff640 10 monclient(hunting): get_auth_request con 0x7f90600a8a40 auth_method 0 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905ffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.686+0000 7f905ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.688+0000 7f9064874640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90601460a0 0x7f90600b8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.689+0000 7f905f7fe640 10 monclient(hunting): handle_auth_done global_id 17999 payload 824 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.689+0000 7f905f7fe640 1 -- 172.21.15.7:0/2112668759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90600a8a40 msgr2=0x7f9060144e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.689+0000 7f905f7fe640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90600a8a40 0x7f9060144e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.689+0000 7f905f7fe640 1 -- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90601460a0 msgr2=0x7f90600b8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.689+0000 7f905f7fe640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90601460a0 0x7f90600b8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.689+0000 7f905f7fe640 10 monclient: _finish_hunting 0 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.689+0000 7f905f7fe640 1 monclient: found mon.noname-c 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.689+0000 7f905f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.689+0000 7f905f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.689+0000 7f905f7fe640 1 -- 172.21.15.7:0/2112668759 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9058009b30 con 0x7f90600b21c0 2024-01-21T03:56:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.689+0000 7f905ffff640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90600a8a40 0x7f9060144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T03:56:27.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.691+0000 7f905f7fe640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 0x7f90600b7b50 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7f905400c7f0 tx=0x7f905400ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:27.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.691+0000 7f905cff9640 10 client.?.objecter ms_handle_connect 0x7f90600b21c0 2024-01-21T03:56:27.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.691+0000 7f905cff9640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:27.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.695+0000 7f905cff9640 1 -- 172.21.15.7:0/2112668759 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9054004220 con 0x7f90600b21c0 2024-01-21T03:56:27.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f905cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:27.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f905cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:56:27.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f905cff9640 10 monclient: dump: 2024-01-21T03:56:27.697 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f905cff9640 10 monclient: _finish_auth 0 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f905cff9640 10 monclient: _check_auth_tickets 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f905cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f905cff9640 1 -- 172.21.15.7:0/2112668759 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9054004380 con 0x7f90600b21c0 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f90662d7640 5 monclient: authenticate success, global_id 17999 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f90662d7640 10 monclient: _renew_subs 2024-01-21T03:56:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f90662d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f90662d7640 1 -- 172.21.15.7:0/2112668759 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90601429e0 con 0x7f90600b21c0 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f905cff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f905cff9640 1 -- 172.21.15.7:0/2112668759 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9054007840 con 0x7f90600b21c0 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f905cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f905cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f905cff9640 10 monclient: dump: 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:27.699 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f905e7fc640 4 set_mon_vals no callback set 2024-01-21T03:56:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f90662d7640 10 client.17999.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f90662d7640 10 monclient: _renew_subs 2024-01-21T03:56:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f90662d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f90662d7640 1 -- 172.21.15.7:0/2112668759 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9060142270 con 0x7f90600b21c0 2024-01-21T03:56:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.696+0000 7f90662d7640 1 librados: init done 2024-01-21T03:56:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.697+0000 7f90662d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.697+0000 7f90662d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.697+0000 7f90662d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.697+0000 7f90662d7640 1 -- 172.21.15.7:0/2112668759 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9060142c20 con 0x7f90600b21c0 2024-01-21T03:56:27.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.720+0000 7f905cff9640 1 -- 172.21.15.7:0/2112668759 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9054007bb0 con 0x7f90600b21c0 2024-01-21T03:56:27.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.720+0000 7f905cff9640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f904c06c8b0 0x7f904c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:27.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.720+0000 7f905ffff640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f904c06c8b0 0x7f904c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:27.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.720+0000 7f905ffff640 10 monclient: get_auth_request con 0x7f904c06c8b0 auth_method 0 2024-01-21T03:56:27.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.720+0000 7f905ffff640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f904c06c8b0 0x7f904c06ed70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9050009f30 tx=0x7f9050009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:27.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.720+0000 7f905cff9640 10 client.17999.objecter ms_handle_connect 0x7f904c06c8b0 2024-01-21T03:56:27.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.991+0000 7f905cff9640 1 -- 172.21.15.7:0/2112668759 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7f9054058380 con 0x7f90600b21c0 2024-01-21T03:56:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.991+0000 7f905cff9640 10 client.17999.objecter ms_dispatch 0x7f9060000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.991+0000 7f905cff9640 3 client.17999.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.991+0000 7f905cff9640 3 client.17999.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.991+0000 7f905cff9640 20 client.17999.objecter dump_active .. 0 homeless 2024-01-21T03:56:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.991+0000 7f905cff9640 1 -- 172.21.15.7:0/2112668759 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f90540546c0 con 0x7f90600b21c0 2024-01-21T03:56:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.991+0000 7f905cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:27.991+0000 7f905cff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:28.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.181+0000 7f90662d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:56:28.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.181+0000 7f90662d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:56:28.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.181+0000 7f90662d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:28.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.181+0000 7f90662d7640 1 -- 172.21.15.7:0/2112668759 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9060142c20 con 0x7f90600b21c0 2024-01-21T03:56:28.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.221+0000 7f905cff9640 1 -- 172.21.15.7:0/2112668759 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1294) v1 ==== 74+0+17680 (secure 0 0 0) 0x7f9060142c20 con 0x7f90600b21c0 2024-01-21T03:56:28.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.221+0000 7f905cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T03:56:28.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.221+0000 7f905cff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:56:28.222 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:56:28.222 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":1294,"fsid":"8de9e7dd-e982-4048-83dd-ee52350dd56d","created":"2024-01-21T03:18:57.607470+0000","modified":"2024-01-21T03:56:25.053760+0000","last_up_change":"2024-01-21T03:19:56.402389+0000","last_in_change":"2024-01-21T03:19:02.952334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":116,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-21T03:19:44.021407+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-21T03:19:59.115939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-01-21T03:21:01.498819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f4767aa-192e-4ec9-addf-a51c0872da7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":1290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6819","nonce":2019292672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6821","nonce":2019292672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6825","nonce":2019292672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":2019292672},{"type":"v1","addr":"172.21.15.7:6823","nonce":2019292672}]},"public_addr":"172.21.15.7:6819/2019292672","cluster_addr":"172.21.15.7:6821/2019292672","heartbeat_back_addr":"172.21.15.7:6825/2019292672","heartbeat_front_addr":"172.21.15.7:6823/2019292672","state":["exists","up"]},{"osd":1,"uuid":"066d7378-561f-4e74-b97b-2ead2fa2625e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":1276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6811","nonce":2176696455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6813","nonce":2176696455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6817","nonce":2176696455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2176696455},{"type":"v1","addr":"172.21.15.7:6815","nonce":2176696455}]},"public_addr":"172.21.15.7:6811/2176696455","cluster_addr":"172.21.15.7:6813/2176696455","heartbeat_back_addr":"172.21.15.7:6817/2176696455","heartbeat_front_addr":"172.21.15.7:6815/2176696455","state":["exists","up"]},{"osd":2,"uuid":"f219649b-4d50-40a8-b74e-5a7f16b3c8c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":1283,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6827","nonce":888884750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6829","nonce":888884750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6833","nonce":888884750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":888884750},{"type":"v1","addr":"172.21.15.7:6831","nonce":888884750}]},"public_addr":"172.21.15.7:6827/888884750","cluster_addr":"172.21.15.7:6829/888884750","heartbeat_back_addr":"172.21.15.7:6833/888884750","heartbeat_front_addr":"172.21.15.7:6831/888884750","state":["exists","up"]},{"osd":3,"uuid":"6379a402-760a-4f07-9491-0918752a18fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":1290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6803","nonce":1182044719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6805","nonce":1182044719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6809","nonce":1182044719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":1182044719},{"type":"v1","addr":"172.21.15.7:6807","nonce":1182044719}]},"public_addr":"172.21.15.7:6803/1182044719","cluster_addr":"172.21.15.7:6805/1182044719","heartbeat_back_addr":"172.21.15.7:6809/1182044719","heartbeat_front_addr":"172.21.15.7:6807/1182044719","state":["exists","up"]},{"osd":4,"uuid":"dc3602e1-b59e-4489-9882-40c914e2a0d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6809","nonce":3975537014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6811","nonce":3975537014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6815","nonce":3975537014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3975537014},{"type":"v1","addr":"172.21.15.177:6813","nonce":3975537014}]},"public_addr":"172.21.15.177:6809/3975537014","cluster_addr":"172.21.15.177:6811/3975537014","heartbeat_back_addr":"172.21.15.177:6815/3975537014","heartbeat_front_addr":"172.21.15.177:6813/3975537014","state":["exists","up"]},{"osd":5,"uuid":"d42afb34-e1ca-4b21-81e9-78d1c8ea454e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6817","nonce":2318383718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6819","nonce":2318383718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6824","nonce":2318383718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2318383718},{"type":"v1","addr":"172.21.15.177:6821","nonce":2318383718}]},"public_addr":"172.21.15.177:6817/2318383718","cluster_addr":"172.21.15.177:6819/2318383718","heartbeat_back_addr":"172.21.15.177:6824/2318383718","heartbeat_front_addr":"172.21.15.177:6821/2318383718","state":["exists","up"]},{"osd":6,"uuid":"1aef2850-8b41-4d39-859e-ed9c4df434b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6801","nonce":1092304531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6803","nonce":1092304531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6807","nonce":1092304531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1092304531},{"type":"v1","addr":"172.21.15.177:6805","nonce":1092304531}]},"public_addr":"172.21.15.177:6801/1092304531","cluster_addr":"172.21.15.177:6803/1092304531","heartbeat_back_addr":"172.21.15.177:6807/1092304531","heartbeat_front_addr":"172.21.15.177:6805/1092304531","state":["exists","up"]},{"osd":7,"uuid":"8157cb10-6ca3-437e-93eb-cf0eb3aa8b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6823","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6825","nonce":4069998673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6827","nonce":4069998673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6831","nonce":4069998673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":4069998673},{"type":"v1","addr":"172.21.15.177:6829","nonce":4069998673}]},"public_addr":"172.21.15.177:6825/4069998673","cluster_addr":"172.21.15.177:6827/4069998673","heartbeat_back_addr":"172.21.15.177:6831/4069998673","heartbeat_front_addr":"172.21.15.177:6829/4069998673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:40.924122+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:41.225506+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:41.727729+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:41.835880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:38.955717+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:38.638101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:39.011972+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T03:19:39.042445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.6","mappings":[{"from":5,"to":0}]},{"pgid":"14.f","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":4,"length":1}]},{"pool":12,"snaps":[{"begin":5,"length":2}]},{"pool":32,"snaps":[{"begin":1,"length":1}]},{"pool":77,"snaps":[{"begin":1,"length":1}]},{"pool":85,"snaps":[{"begin":1,"length":1}]},{"pool":98,"snaps":[{"begin":7,"length":2}]},{"pool":103,"snaps":[{"begin":1,"length":2}]},{"pool":120,"snaps":[{"begin":3,"length":1}]},{"pool":123,"snaps":[{"begin":2,"length":2}]},{"pool":126,"snaps":[{"begin":2,"length":3}]},{"pool":130,"snaps":[{"begin":2,"length":2}]},{"pool":131,"snaps":[{"begin":3,"length":2}]},{"pool":138,"snaps":[{"begin":2,"length":1}]},{"pool":142,"snaps":[{"begin":2,"length":4}]},{"pool":143,"snaps":[{"begin":2,"length":1}]},{"pool":145,"snaps":[{"begin":1,"length":4}]},{"pool":147,"snaps":[{"begin":2,"length":6}]},{"pool":188,"snaps":[{"begin":7,"length":1}]},{"pool":198,"snaps":[{"begin":8,"length":1}]},{"pool":203,"snaps":[{"begin":10,"length":1}]},{"pool":204,"snaps":[{"begin":11,"length":2}]},{"pool":207,"snaps":[{"begin":14,"length":1}]},{"pool":250,"snaps":[{"begin":2,"length":1}]},{"pool":251,"snaps":[{"begin":3,"length":1}]},{"pool":254,"snaps":[{"begin":4,"length":1}]},{"pool":258,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T03:56:28.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 10 librados: watch_flush enter 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 10 librados: watch_flush exit 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 20 client.17999.objecter shutdown clearing up homeless session... 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 10 client.17999.objecter successfully canceled tick 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 1 -- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f904c06c8b0 msgr2=0x7f904c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f904c06c8b0 0x7f904c06ed70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9050009f30 tx=0x7f9050009190 comp rx=0 tx=0).stop 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 10 monclient: shutdown 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 1 -- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 msgr2=0x7f90600b7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 0x7f90600b7b50 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7f905400c7f0 tx=0x7f905400ccb0 comp rx=0 tx=0).stop 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 1 -- 172.21.15.7:0/2112668759 shutdown_connections 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f904c06c8b0 0x7f904c06ed70 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90601460a0 0x7f90600b8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90600b21c0 0x7f90600b7b50 unknown :-1 s=CLOSED pgs=3149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:28.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 1 --2- 172.21.15.7:0/2112668759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90600a8a40 0x7f9060144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 1 -- 172.21.15.7:0/2112668759 >> 172.21.15.7:0/2112668759 conn(0x7f9060066f50 msgr2=0x7f906013cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 1 -- 172.21.15.7:0/2112668759 shutdown_connections 2024-01-21T03:56:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.223+0000 7f90662d7640 1 -- 172.21.15.7:0/2112668759 wait complete. 2024-01-21T03:56:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:28.224+0000 7f90662d7640 1 librados: shutdown 2024-01-21T03:56:29.237 INFO:tasks.ceph:Scrubbing osd.0 2024-01-21T03:56:29.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-01-21T03:56:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.343+0000 7f4525a90640 10 monclient: get_monmap_and_config 2024-01-21T03:56:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.343+0000 7f4525a90640 10 monclient: build_initial_monmap 2024-01-21T03:56:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.343+0000 7f4525a90640 10 monclient: monmap: 2024-01-21T03:56:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:29.344406+0000 2024-01-21T03:56:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:29.344406+0000 2024-01-21T03:56:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:29.345 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 1 Processor -- start 2024-01-21T03:56:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 1 -- start start 2024-01-21T03:56:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 10 monclient: init 2024-01-21T03:56:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-01-21T03:56:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45200a8b20 0x7f4520145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 10 monclient(hunting): picked mon.noname-b con 0x7f45200a8b20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45200a9d10 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f45200b2310 0x7f452013f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 10 monclient(hunting): picked mon.noname-c con 0x7f45200b2310 addr v2:172.21.15.177:3300/0 2024-01-21T03:56:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45201456f0 con 0x7f45200a9d10 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45200a98c0 con 0x7f45200b2310 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45201460c0 con 0x7f45200a8b20 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.344+0000 7f4525a90640 10 monclient(hunting): authenticate will time out at 3535.660557s 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.345+0000 7f451ffff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f45200b2310 0x7f452013f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.345+0000 7f451f7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45200a8b20 0x7f4520145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.345+0000 7f451f7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45200a8b20 0x7f4520145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37000/0 (socket says 172.21.15.7:37000) 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.345+0000 7f451f7fe640 1 -- 172.21.15.7:0/1955837367 learned_addr learned my addr 172.21.15.7:0/1955837367 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.345+0000 7f451f7fe640 10 monclient(hunting): get_auth_request con 0x7f45200a8b20 auth_method 0 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.345+0000 7f451f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.345+0000 7f451f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.345+0000 7f451f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.346+0000 7f451ffff640 10 monclient(hunting): get_auth_request con 0x7f45200b2310 auth_method 0 2024-01-21T03:56:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.346+0000 7f451ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.346+0000 7f451ffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.346+0000 7f451ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.346+0000 7f451f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.346+0000 7f451f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.346+0000 7f451f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.346+0000 7f451ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.346+0000 7f451ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.346+0000 7f451ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.348+0000 7f451ffff640 10 monclient(hunting): handle_auth_done global_id 23146 payload 274 2024-01-21T03:56:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.348+0000 7f451ffff640 1 -- 172.21.15.7:0/1955837367 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45200a8b20 msgr2=0x7f4520145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.348+0000 7f451ffff640 1 --2- 172.21.15.7:0/1955837367 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45200a8b20 0x7f4520145b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.348+0000 7f451ffff640 1 -- 172.21.15.7:0/1955837367 >> v1:172.21.15.177:6789/0 conn(0x7f45200a9d10 legacy=0x7f452013f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.348+0000 7f451ffff640 10 monclient: _finish_hunting 0 2024-01-21T03:56:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.348+0000 7f451ffff640 1 monclient: found mon.noname-c 2024-01-21T03:56:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.348+0000 7f451ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.348+0000 7f451ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:56:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.348+0000 7f451ffff640 1 -- 172.21.15.7:0/1955837367 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4520146230 con 0x7f45200b2310 2024-01-21T03:56:29.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.351+0000 7f451ffff640 1 --2- 172.21.15.7:0/1955837367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45200b2310 0x7f452013f790 secure :-1 s=READY pgs=5807 cs=0 l=1 rev1=1 crypto rx=0x7f4518009d90 tx=0x7f451802be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d01bbaa7d88b5285 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:29.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.356+0000 7f451e7fc640 1 -- 172.21.15.7:0/1955837367 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451802cbd0 con 0x7f45200b2310 2024-01-21T03:56:29.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.356+0000 7f451e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:29.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.356+0000 7f451e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.356+0000 7f451e7fc640 10 monclient: dump: 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f451e7fc640 10 monclient: _finish_auth 0 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f451e7fc640 10 monclient: _check_auth_tickets 2024-01-21T03:56:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f451e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f451e7fc640 1 -- 172.21.15.7:0/1955837367 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451802cd30 con 0x7f45200b2310 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 5 monclient: authenticate success, global_id 23146 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f451e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f451e7fc640 1 -- 172.21.15.7:0/1955837367 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451802cea0 con 0x7f45200b2310 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 4 set_mon_vals no callback set 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f451e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f451e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f451e7fc640 10 monclient: dump: 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:29.360 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:29.360 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:29.360 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:29.360 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 10 monclient: shutdown 2024-01-21T03:56:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 1 -- 172.21.15.7:0/1955837367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45200b2310 msgr2=0x7f452013f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 1 --2- 172.21.15.7:0/1955837367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45200b2310 0x7f452013f790 secure :-1 s=READY pgs=5807 cs=0 l=1 rev1=1 crypto rx=0x7f4518009d90 tx=0x7f451802be20 comp rx=0 tx=0).stop 2024-01-21T03:56:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 1 -- 172.21.15.7:0/1955837367 shutdown_connections 2024-01-21T03:56:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 1 --2- 172.21.15.7:0/1955837367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45200b2310 0x7f452013f790 unknown :-1 s=CLOSED pgs=5807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 1 --2- 172.21.15.7:0/1955837367 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45200a8b20 0x7f4520145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 1 -- 172.21.15.7:0/1955837367 >> 172.21.15.7:0/1955837367 conn(0x7f4520066fb0 msgr2=0x7f45200b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 1 -- 172.21.15.7:0/1955837367 shutdown_connections 2024-01-21T03:56:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.357+0000 7f4525a90640 1 -- 172.21.15.7:0/1955837367 wait complete. 2024-01-21T03:56:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.358+0000 7f4525a90640 10 monclient: build_initial_monmap 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.358+0000 7f4525a90640 10 monclient: monmap: 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.358+0000 7f4525a90640 1 librados: starting msgr at 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.358+0000 7f4525a90640 1 librados: starting objecter 2024-01-21T03:56:29.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.358+0000 7f4525a90640 1 Processor -- start 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 1 -- start start 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 1 librados: setting wanted keys 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 1 librados: calling monclient init 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 10 monclient: init 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45200a8b20 0x7f4520142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 10 monclient(hunting): picked mon.noname-a con 0x7f45200a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45200a9d10 0x7f4520132c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 10 monclient(hunting): picked mon.noname-c con 0x7f45200a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45200b2310 0x7f4520133150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 10 monclient(hunting): picked mon.noname-b con 0x7f45200b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45201383f0 con 0x7f45200a8b20 2024-01-21T03:56:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45201395f0 con 0x7f45200b2310 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4520139860 con 0x7f45200a9d10 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.359+0000 7f4525a90640 10 monclient(hunting): authenticate will time out at 3535.675386s 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451effd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45200a9d10 0x7f4520132c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45200a8b20 0x7f4520142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45200b2310 0x7f4520133150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451effd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45200a9d10 0x7f4520132c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37006/0 (socket says 172.21.15.7:37006) 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451effd640 1 -- 172.21.15.7:0/3978368909 learned_addr learned my addr 172.21.15.7:0/3978368909 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451effd640 10 monclient(hunting): get_auth_request con 0x7f45200a9d10 auth_method 0 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451effd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451ffff640 10 monclient(hunting): get_auth_request con 0x7f45200b2310 auth_method 0 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451ffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451f7fe640 10 monclient(hunting): get_auth_request con 0x7f45200a8b20 auth_method 0 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.360+0000 7f451f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.361+0000 7f451effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.361+0000 7f451effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.361+0000 7f451effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.361+0000 7f451ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.361+0000 7f451ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.361+0000 7f451ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.361+0000 7f451f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.361+0000 7f451f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.361+0000 7f451f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:29.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.363+0000 7f451ffff640 10 monclient(hunting): handle_auth_done global_id 23149 payload 824 2024-01-21T03:56:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.363+0000 7f451ffff640 1 -- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45200a9d10 msgr2=0x7f4520132c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.363+0000 7f451ffff640 1 --2- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45200a9d10 0x7f4520132c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.363+0000 7f451ffff640 1 -- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45200a8b20 msgr2=0x7f4520142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.363+0000 7f451ffff640 1 --2- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45200a8b20 0x7f4520142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.363+0000 7f451ffff640 10 monclient: _finish_hunting 0 2024-01-21T03:56:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.363+0000 7f451ffff640 1 monclient: found mon.noname-b 2024-01-21T03:56:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.363+0000 7f451ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.363+0000 7f451ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:56:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.363+0000 7f451ffff640 1 -- 172.21.15.7:0/3978368909 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4518009a70 con 0x7f45200b2310 2024-01-21T03:56:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.363+0000 7f451effd640 1 --2- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45200a9d10 0x7f4520132c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T03:56:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.366+0000 7f451ffff640 1 --2- 172.21.15.7:0/3978368909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45200b2310 0x7f4520133150 secure :-1 s=READY pgs=5808 cs=0 l=1 rev1=1 crypto rx=0x7f45200b89d0 tx=0x7f4518004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.366+0000 7f4507fff640 10 client.?.objecter ms_handle_connect 0x7f45200b2310 2024-01-21T03:56:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.366+0000 7f4507fff640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:29.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4507fff640 1 -- 172.21.15.7:0/3978368909 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451802cea0 con 0x7f45200b2310 2024-01-21T03:56:29.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4507fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:29.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4507fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:29.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4507fff640 10 monclient: dump: 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4507fff640 10 monclient: _finish_auth 0 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4507fff640 10 monclient: _check_auth_tickets 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4507fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4507fff640 1 -- 172.21.15.7:0/3978368909 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451803e650 con 0x7f45200b2310 2024-01-21T03:56:29.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4525a90640 5 monclient: authenticate success, global_id 23149 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4525a90640 10 monclient: _renew_subs 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4525a90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4525a90640 1 -- 172.21.15.7:0/3978368909 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4520133690 con 0x7f45200b2310 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4507fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4507fff640 1 -- 172.21.15.7:0/3978368909 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451803ea00 con 0x7f45200b2310 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4507fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4507fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f4507fff640 10 monclient: dump: 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:29.372 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:29.373 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:29.373 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:29.373 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:29.373 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:29.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.369+0000 7f451d7fa640 4 set_mon_vals no callback set 2024-01-21T03:56:29.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.370+0000 7f4525a90640 10 client.23149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:29.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.370+0000 7f4525a90640 10 monclient: _renew_subs 2024-01-21T03:56:29.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.370+0000 7f4525a90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:29.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.370+0000 7f4525a90640 1 -- 172.21.15.7:0/3978368909 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4520138ae0 con 0x7f45200b2310 2024-01-21T03:56:29.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.370+0000 7f4525a90640 1 librados: init done 2024-01-21T03:56:29.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.370+0000 7f4525a90640 10 client.23149.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:29.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.370+0000 7f4525a90640 15 client.23149.objecter _session_command_op_assign -1 1 2024-01-21T03:56:29.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.370+0000 7f4525a90640 20 client.23149.objecter _get_session osd=-1 returning homeless 2024-01-21T03:56:29.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.370+0000 7f4525a90640 10 client.23149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:29.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.370+0000 7f4525a90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:29.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.370+0000 7f4525a90640 1 -- 172.21.15.7:0/3978368909 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f45200a9110 con 0x7f45200b2310 2024-01-21T03:56:29.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.390+0000 7f4507fff640 1 -- 172.21.15.7:0/3978368909 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f451803b050 con 0x7f45200b2310 2024-01-21T03:56:29.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.390+0000 7f4507fff640 1 --2- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f450c06c8b0 0x7f450c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:29.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.390+0000 7f451f7fe640 1 --2- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f450c06c8b0 0x7f450c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:29.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.390+0000 7f451f7fe640 10 monclient: get_auth_request con 0x7f450c06c8b0 auth_method 0 2024-01-21T03:56:29.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.390+0000 7f4507fff640 10 client.23149.objecter ms_handle_connect 0x7f450c06c8b0 2024-01-21T03:56:29.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.390+0000 7f451f7fe640 1 --2- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f450c06c8b0 0x7f450c06ed70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4510009bd0 tx=0x7f4510009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:29.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 1 -- 172.21.15.7:0/3978368909 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7f451807adf0 con 0x7f45200b2310 2024-01-21T03:56:29.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 10 client.23149.objecter ms_dispatch 0x7f4520000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:29.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 3 client.23149.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:29.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 3 client.23149.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:29.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 10 client.23149.objecter checking command 1 2024-01-21T03:56:29.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 1 --2- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] conn(0x7f450c0746a0 0x7f450c076b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:29.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 20 client.23149.objecter _get_session s=0x7f450c073fd0 osd=0 3 2024-01-21T03:56:29.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 20 client.23149.objecter put_session s=0x7f450c073fd0 osd=0 3 2024-01-21T03:56:29.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 15 client.23149.objecter _session_command_op_remove -1 1 2024-01-21T03:56:29.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 20 client.23149.objecter _get_session s=0x7f450c073fd0 osd=0 3 2024-01-21T03:56:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 20 client.23149.objecter get_session s=0x7f450c073fd0 osd=0 3 2024-01-21T03:56:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 15 client.23149.objecter _session_command_op_assign 0 1 2024-01-21T03:56:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 20 client.23149.objecter put_session s=0x7f450c073fd0 osd=0 4 2024-01-21T03:56:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 10 client.23149.objecter _send_command 1 2024-01-21T03:56:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 1 -- 172.21.15.7:0/3978368909 --> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f450c077210 con 0x7f450c0746a0 2024-01-21T03:56:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 20 client.23149.objecter dump_active .. 0 homeless 2024-01-21T03:56:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 1 -- 172.21.15.7:0/3978368909 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=1294) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4518077130 con 0x7f45200b2310 2024-01-21T03:56:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f4507fff640 10 monclient: handle_get_version_reply finishing 1 version 1294 2024-01-21T03:56:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.392+0000 7f451effd640 1 --2- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] conn(0x7f450c0746a0 0x7f450c076b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.393+0000 7f451effd640 10 monclient: get_auth_request con 0x7f450c0746a0 auth_method 0 2024-01-21T03:56:29.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.395+0000 7f451effd640 1 --2- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] conn(0x7f450c0746a0 0x7f450c076b40 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:29.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.395+0000 7f4507fff640 10 client.23149.objecter ms_handle_connect 0x7f450c0746a0 2024-01-21T03:56:29.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.421+0000 7f4507fff640 1 -- 172.21.15.7:0/3978368909 <== osd.0 v2:172.21.15.7:6818/2019292672 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f450c077210 con 0x7f450c0746a0 2024-01-21T03:56:29.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.421+0000 7f4507fff640 10 client.23149.objecter ms_dispatch 0x7f4520000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:56:29.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.421+0000 7f4507fff640 10 client.23149.objecter _finish_command 1 = system:0 2024-01-21T03:56:29.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.421+0000 7f4507fff640 20 client.23149.objecter put_session s=0x7f450c073fd0 osd=0 4 2024-01-21T03:56:29.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.421+0000 7f4507fff640 15 client.23149.objecter _session_command_op_remove 0 1 2024-01-21T03:56:29.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.434+0000 7f4525a90640 10 client.23149.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T03:56:29.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.434+0000 7f4525a90640 15 client.23149.objecter _session_command_op_assign -1 2 2024-01-21T03:56:29.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.434+0000 7f4525a90640 20 client.23149.objecter _get_session s=0x7f450c073fd0 osd=0 3 2024-01-21T03:56:29.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.434+0000 7f4525a90640 20 client.23149.objecter put_session s=0x7f450c073fd0 osd=0 3 2024-01-21T03:56:29.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.434+0000 7f4525a90640 20 client.23149.objecter _get_session s=0x7f450c073fd0 osd=0 3 2024-01-21T03:56:29.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.434+0000 7f4525a90640 15 client.23149.objecter _session_command_op_remove -1 2 2024-01-21T03:56:29.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.434+0000 7f4525a90640 20 client.23149.objecter get_session s=0x7f450c073fd0 osd=0 3 2024-01-21T03:56:29.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.434+0000 7f4525a90640 15 client.23149.objecter _session_command_op_assign 0 2 2024-01-21T03:56:29.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.434+0000 7f4525a90640 20 client.23149.objecter put_session s=0x7f450c073fd0 osd=0 4 2024-01-21T03:56:29.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.434+0000 7f4525a90640 10 client.23149.objecter _send_command 2 2024-01-21T03:56:29.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.434+0000 7f4525a90640 1 -- 172.21.15.7:0/3978368909 --> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4520139d80 con 0x7f450c0746a0 2024-01-21T03:56:29.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.450+0000 7f4507fff640 1 -- 172.21.15.7:0/3978368909 <== osd.0 v2:172.21.15.7:6818/2019292672 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4520139d80 con 0x7f450c0746a0 2024-01-21T03:56:29.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.450+0000 7f4507fff640 10 client.23149.objecter ms_dispatch 0x7f4520000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:56:29.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.450+0000 7f4507fff640 10 client.23149.objecter _finish_command 2 = system:0 2024-01-21T03:56:29.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.450+0000 7f4507fff640 20 client.23149.objecter put_session s=0x7f450c073fd0 osd=0 4 2024-01-21T03:56:29.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.450+0000 7f4507fff640 15 client.23149.objecter _session_command_op_remove 0 2 2024-01-21T03:56:29.451 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-01-21T03:56:29.451 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T03:56:29.452 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-01-21T03:56:29.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 10 librados: watch_flush enter 2024-01-21T03:56:29.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 10 librados: watch_flush exit 2024-01-21T03:56:29.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 10 client.23149.objecter close_session for osd.0 2024-01-21T03:56:29.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 1 -- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] conn(0x7f450c0746a0 msgr2=0x7f450c076b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:29.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 1 --2- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:6818/2019292672,v1:172.21.15.7:6819/2019292672] conn(0x7f450c0746a0 0x7f450c076b40 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:29.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 20 client.23149.objecter put_session s=0x7f450c073fd0 osd=0 1 2024-01-21T03:56:29.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 20 client.23149.objecter shutdown clearing up homeless session... 2024-01-21T03:56:29.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 10 client.23149.objecter successfully canceled tick 2024-01-21T03:56:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 1 -- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f450c06c8b0 msgr2=0x7f450c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 1 --2- 172.21.15.7:0/3978368909 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f450c06c8b0 0x7f450c06ed70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4510009bd0 tx=0x7f4510009240 comp rx=0 tx=0).stop 2024-01-21T03:56:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 10 monclient: shutdown 2024-01-21T03:56:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 1 -- 172.21.15.7:0/3978368909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45200b2310 msgr2=0x7f4520133150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 1 --2- 172.21.15.7:0/3978368909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45200b2310 0x7f4520133150 secure :-1 s=READY pgs=5808 cs=0 l=1 rev1=1 crypto rx=0x7f45200b89d0 tx=0x7f4518004a40 comp rx=0 tx=0).stop 2024-01-21T03:56:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f451ffff640 1 -- 172.21.15.7:0/3978368909 reap_dead start 2024-01-21T03:56:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 1 -- 172.21.15.7:0/3978368909 shutdown_connections 2024-01-21T03:56:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 1 -- 172.21.15.7:0/3978368909 >> 172.21.15.7:0/3978368909 conn(0x7f4520066fb0 msgr2=0x7f45200b6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 1 -- 172.21.15.7:0/3978368909 shutdown_connections 2024-01-21T03:56:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 1 -- 172.21.15.7:0/3978368909 wait complete. 2024-01-21T03:56:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.451+0000 7f4525a90640 1 librados: shutdown 2024-01-21T03:56:29.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-01-21T03:56:29.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.562+0000 7f9adfada640 10 monclient: get_monmap_and_config 2024-01-21T03:56:29.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.562+0000 7f9adfada640 10 monclient: build_initial_monmap 2024-01-21T03:56:29.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.562+0000 7f9adfada640 10 monclient: monmap: 2024-01-21T03:56:29.563 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:29.563 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:29.563 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:29.563753+0000 2024-01-21T03:56:29.563 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:29.563753+0000 2024-01-21T03:56:29.563 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:29.563 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:29.563 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:29.564 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:29.564 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:29.564 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:29.564 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:29.564 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:29.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.563+0000 7f9adfada640 1 Processor -- start 2024-01-21T03:56:29.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.563+0000 7f9adfada640 1 -- start start 2024-01-21T03:56:29.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.563+0000 7f9adfada640 10 monclient: init 2024-01-21T03:56:29.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:29.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-01-21T03:56:29.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9ad80a8ab0 0x7f9ad813efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad80a8ab0 addr v2:172.21.15.177:3300/0 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ad813f500 0x7f9ad81422e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad813f500 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ad8142820 addr v1:172.21.15.7:6789/0 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad80aaa20 con 0x7f9ad8142820 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad80aad00 con 0x7f9ad80a8ab0 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad80aafe0 con 0x7f9ad813f500 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9adfada640 10 monclient(hunting): authenticate will time out at 3535.879928s 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9add075640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ad813f500 0x7f9ad81422e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9add876640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9ad80a8ab0 0x7f9ad813efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9ade077640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9ad8142820 0x7f9ad81455c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56402/0 (socket says 172.21.15.7:56402) 2024-01-21T03:56:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.564+0000 7f9ade077640 1 -- 172.21.15.7:0/302364082 learned_addr learned my addr 172.21.15.7:0/302364082 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add075640 10 monclient(hunting): get_auth_request con 0x7f9ad813f500 auth_method 0 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add075640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add075640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add075640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add876640 10 monclient(hunting): get_auth_request con 0x7f9ad80a8ab0 auth_method 0 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add876640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add876640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add876640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add876640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add876640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add876640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add075640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add075640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.565+0000 7f9add075640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:29.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.567+0000 7f9add876640 10 monclient(hunting): handle_auth_done global_id 23152 payload 274 2024-01-21T03:56:29.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.567+0000 7f9add876640 1 -- 172.21.15.7:0/302364082 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ad813f500 msgr2=0x7f9ad81422e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:29.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.567+0000 7f9add876640 1 --2- 172.21.15.7:0/302364082 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ad813f500 0x7f9ad81422e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:29.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.567+0000 7f9add876640 1 -- 172.21.15.7:0/302364082 >> v1:172.21.15.7:6789/0 conn(0x7f9ad8142820 legacy=0x7f9ad81455c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:29.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.567+0000 7f9add876640 10 monclient: _finish_hunting 0 2024-01-21T03:56:29.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.567+0000 7f9add876640 1 monclient: found mon.noname-c 2024-01-21T03:56:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.567+0000 7f9add876640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.567+0000 7f9add876640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:56:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.567+0000 7f9add876640 1 -- 172.21.15.7:0/302364082 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad80a91b0 con 0x7f9ad80a8ab0 2024-01-21T03:56:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.569+0000 7f9add876640 1 --2- 172.21.15.7:0/302364082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ad80a8ab0 0x7f9ad813efc0 secure :-1 s=READY pgs=5809 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40097a0 tx=0x7f9ad402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df7354441b78deff server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adc874640 1 -- 172.21.15.7:0/302364082 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4031830 con 0x7f9ad80a8ab0 2024-01-21T03:56:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adc874640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adc874640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:56:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adc874640 10 monclient: dump: 2024-01-21T03:56:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adc874640 10 monclient: _finish_auth 0 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adc874640 10 monclient: _check_auth_tickets 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adc874640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adc874640 1 -- 172.21.15.7:0/302364082 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad4031990 con 0x7f9ad80a8ab0 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adfada640 5 monclient: authenticate success, global_id 23152 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adfada640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adc874640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adc874640 1 -- 172.21.15.7:0/302364082 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4031d20 con 0x7f9ad80a8ab0 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adfada640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:29.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adfada640 4 set_mon_vals no callback set 2024-01-21T03:56:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adc874640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adc874640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.572+0000 7f9adc874640 10 monclient: dump: 2024-01-21T03:56:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:29.575 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.573+0000 7f9adfada640 10 monclient: shutdown 2024-01-21T03:56:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.573+0000 7f9adfada640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.573+0000 7f9adfada640 1 -- 172.21.15.7:0/302364082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ad80a8ab0 msgr2=0x7f9ad813efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.573+0000 7f9adfada640 1 --2- 172.21.15.7:0/302364082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ad80a8ab0 0x7f9ad813efc0 secure :-1 s=READY pgs=5809 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40097a0 tx=0x7f9ad402b8d0 comp rx=0 tx=0).stop 2024-01-21T03:56:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.573+0000 7f9adfada640 1 -- 172.21.15.7:0/302364082 shutdown_connections 2024-01-21T03:56:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.573+0000 7f9adfada640 1 --2- 172.21.15.7:0/302364082 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ad813f500 0x7f9ad81422e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.573+0000 7f9adfada640 1 --2- 172.21.15.7:0/302364082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ad80a8ab0 0x7f9ad813efc0 unknown :-1 s=CLOSED pgs=5809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.573+0000 7f9adfada640 1 -- 172.21.15.7:0/302364082 >> 172.21.15.7:0/302364082 conn(0x7f9ad8066f50 msgr2=0x7f9ad80b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.573+0000 7f9adfada640 1 -- 172.21.15.7:0/302364082 shutdown_connections 2024-01-21T03:56:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.573+0000 7f9adfada640 1 -- 172.21.15.7:0/302364082 wait complete. 2024-01-21T03:56:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.574+0000 7f9adfada640 10 monclient: build_initial_monmap 2024-01-21T03:56:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.574+0000 7f9adfada640 10 monclient: monmap: 2024-01-21T03:56:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.574+0000 7f9adfada640 1 librados: starting msgr at 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.574+0000 7f9adfada640 1 librados: starting objecter 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.574+0000 7f9adfada640 1 Processor -- start 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.574+0000 7f9adfada640 1 -- start start 2024-01-21T03:56:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.574+0000 7f9adfada640 1 librados: setting wanted keys 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.574+0000 7f9adfada640 1 librados: calling monclient init 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.574+0000 7f9adfada640 10 monclient: init 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ad80a8ab0 0x7f9ad80b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad80a8ab0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ad813f500 0x7f9ad80ac600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad813f500 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ad8142820 0x7f9ad80ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad8142820 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad80ac120 con 0x7f9ad8142820 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad80ae5e0 con 0x7f9ad813f500 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad80ad370 con 0x7f9ad80a8ab0 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9adfada640 10 monclient(hunting): authenticate will time out at 3535.891085s 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9add876640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ad80a8ab0 0x7f9ad80b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.575+0000 7f9ade077640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ad8142820 0x7f9ad80ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add876640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ad80a8ab0 0x7f9ad80b0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37024/0 (socket says 172.21.15.7:37024) 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add876640 1 -- 172.21.15.7:0/2193312969 learned_addr learned my addr 172.21.15.7:0/2193312969 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add876640 10 monclient(hunting): get_auth_request con 0x7f9ad80a8ab0 auth_method 0 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add876640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add876640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add876640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add075640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ad813f500 0x7f9ad80ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9ade077640 10 monclient(hunting): get_auth_request con 0x7f9ad8142820 auth_method 0 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9ade077640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9ade077640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9ade077640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add075640 10 monclient(hunting): get_auth_request con 0x7f9ad813f500 auth_method 0 2024-01-21T03:56:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add075640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add075640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add075640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add876640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add876640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add876640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9ade077640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9ade077640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9ade077640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add075640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add075640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.576+0000 7f9add075640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.579+0000 7f9ade077640 10 monclient(hunting): handle_auth_done global_id 15843 payload 824 2024-01-21T03:56:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.579+0000 7f9ade077640 1 -- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ad80a8ab0 msgr2=0x7f9ad80b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.579+0000 7f9ade077640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ad80a8ab0 0x7f9ad80b0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.579+0000 7f9ade077640 1 -- 172.21.15.7:0/2193312969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ad813f500 msgr2=0x7f9ad80ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.579+0000 7f9ade077640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ad813f500 0x7f9ad80ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.579+0000 7f9ade077640 10 monclient: _finish_hunting 0 2024-01-21T03:56:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.579+0000 7f9ade077640 1 monclient: found mon.noname-a 2024-01-21T03:56:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.579+0000 7f9ade077640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.579+0000 7f9ade077640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:56:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.579+0000 7f9ade077640 1 -- 172.21.15.7:0/2193312969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad4009400 con 0x7f9ad8142820 2024-01-21T03:56:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.579+0000 7f9add876640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ad80a8ab0 0x7f9ad80b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T03:56:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.579+0000 7f9add075640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ad813f500 0x7f9ad80ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T03:56:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.581+0000 7f9ade077640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ad8142820 0x7f9ad80ace30 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000caa0 tx=0x7f9ad000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:29.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.581+0000 7f9ace7fc640 10 client.?.objecter ms_handle_connect 0x7f9ad8142820 2024-01-21T03:56:29.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.581+0000 7f9ace7fc640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9ace7fc640 1 -- 172.21.15.7:0/2193312969 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad000ab40 con 0x7f9ad8142820 2024-01-21T03:56:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9ace7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9ace7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:56:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9ace7fc640 10 monclient: dump: 2024-01-21T03:56:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9ace7fc640 10 monclient: _finish_auth 0 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9ace7fc640 10 monclient: _check_auth_tickets 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9ace7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9ace7fc640 1 -- 172.21.15.7:0/2193312969 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad000aca0 con 0x7f9ad8142820 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9ace7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9ace7fc640 1 -- 172.21.15.7:0/2193312969 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0012600 con 0x7f9ad8142820 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9adfada640 5 monclient: authenticate success, global_id 15843 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9adfada640 10 monclient: _renew_subs 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9adfada640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9adfada640 1 -- 172.21.15.7:0/2193312969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad80ad5f0 con 0x7f9ad8142820 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9acffff640 4 set_mon_vals no callback set 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9ace7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9ace7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:56:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9ace7fc640 10 monclient: dump: 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9adfada640 10 client.15843.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9adfada640 10 monclient: _renew_subs 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9adfada640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9adfada640 1 -- 172.21.15.7:0/2193312969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad81d7900 con 0x7f9ad8142820 2024-01-21T03:56:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.585+0000 7f9adfada640 1 librados: init done 2024-01-21T03:56:29.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.586+0000 7f9adfada640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:29.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.586+0000 7f9adfada640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:29.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.586+0000 7f9adfada640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:29.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.586+0000 7f9adfada640 1 -- 172.21.15.7:0/2193312969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad80abf80 con 0x7f9ad8142820 2024-01-21T03:56:29.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.606+0000 7f9ace7fc640 1 -- 172.21.15.7:0/2193312969 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9ad0012760 con 0x7f9ad8142820 2024-01-21T03:56:29.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.606+0000 7f9ace7fc640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f9abc06c8b0 0x7f9abc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:29.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.606+0000 7f9add876640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f9abc06c8b0 0x7f9abc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:29.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.606+0000 7f9add876640 10 monclient: get_auth_request con 0x7f9abc06c8b0 auth_method 0 2024-01-21T03:56:29.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.607+0000 7f9add876640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f9abc06c8b0 0x7f9abc06ed70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9ad402b910 tx=0x7f9ad403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:29.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.607+0000 7f9ace7fc640 10 client.15843.objecter ms_handle_connect 0x7f9abc06c8b0 2024-01-21T03:56:29.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.608+0000 7f9ace7fc640 1 -- 172.21.15.7:0/2193312969 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7f9ad00586b0 con 0x7f9ad8142820 2024-01-21T03:56:29.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.608+0000 7f9ace7fc640 10 client.15843.objecter ms_dispatch 0x7f9ad8000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:29.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.608+0000 7f9ace7fc640 3 client.15843.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:29.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.608+0000 7f9ace7fc640 3 client.15843.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:29.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.608+0000 7f9ace7fc640 20 client.15843.objecter dump_active .. 0 homeless 2024-01-21T03:56:29.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.858+0000 7f9ace7fc640 1 -- 172.21.15.7:0/2193312969 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9ad00549f0 con 0x7f9ad8142820 2024-01-21T03:56:29.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.858+0000 7f9ace7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:29.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:29.858+0000 7f9ace7fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:30.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.036+0000 7f9adfada640 1 -- 172.21.15.7:0/2193312969 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ad813fb80 con 0x7f9abc06c8b0 2024-01-21T03:56:30.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.036+0000 7f9ace7fc640 1 -- 172.21.15.7:0/2193312969 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9ad403a420 con 0x7f9abc06c8b0 2024-01-21T03:56:30.037 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 0 to deep-scrub 2024-01-21T03:56:30.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 10 librados: watch_flush enter 2024-01-21T03:56:30.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 10 librados: watch_flush exit 2024-01-21T03:56:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 20 client.15843.objecter shutdown clearing up homeless session... 2024-01-21T03:56:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 10 client.15843.objecter successfully canceled tick 2024-01-21T03:56:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 1 -- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f9abc06c8b0 msgr2=0x7f9abc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f9abc06c8b0 0x7f9abc06ed70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9ad402b910 tx=0x7f9ad403c040 comp rx=0 tx=0).stop 2024-01-21T03:56:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 10 monclient: shutdown 2024-01-21T03:56:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 1 -- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ad8142820 msgr2=0x7f9ad80ace30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ad8142820 0x7f9ad80ace30 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000caa0 tx=0x7f9ad000cf60 comp rx=0 tx=0).stop 2024-01-21T03:56:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 1 -- 172.21.15.7:0/2193312969 shutdown_connections 2024-01-21T03:56:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f9abc06c8b0 0x7f9abc06ed70 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ad8142820 0x7f9ad80ace30 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ad813f500 0x7f9ad80ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 1 --2- 172.21.15.7:0/2193312969 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9ad80a8ab0 0x7f9ad80b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 1 -- 172.21.15.7:0/2193312969 >> 172.21.15.7:0/2193312969 conn(0x7f9ad8066f50 msgr2=0x7f9ad80b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:30.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 1 -- 172.21.15.7:0/2193312969 shutdown_connections 2024-01-21T03:56:30.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.038+0000 7f9adfada640 1 -- 172.21.15.7:0/2193312969 wait complete. 2024-01-21T03:56:30.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.039+0000 7f9adfada640 1 librados: shutdown 2024-01-21T03:56:30.050 INFO:tasks.ceph:Scrubbing osd.1 2024-01-21T03:56:30.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-01-21T03:56:30.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.156+0000 7f021d2a2640 10 monclient: get_monmap_and_config 2024-01-21T03:56:30.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.156+0000 7f021d2a2640 10 monclient: build_initial_monmap 2024-01-21T03:56:30.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.156+0000 7f021d2a2640 10 monclient: monmap: 2024-01-21T03:56:30.157 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:30.157 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:30.157 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:30.157460+0000 2024-01-21T03:56:30.157 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:30.157460+0000 2024-01-21T03:56:30.157 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:30.157 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.157 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:30.157 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:30.157 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:30.158 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:30.158 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:30.158 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 1 Processor -- start 2024-01-21T03:56:30.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 1 -- start start 2024-01-21T03:56:30.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 10 monclient: init 2024-01-21T03:56:30.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:30.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-01-21T03:56:30.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02180a8a40 addr v1:172.21.15.7:6789/0 2024-01-21T03:56:30.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f02180b2210 0x7f021813f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 10 monclient(hunting): picked mon.noname-c con 0x7f02180b2210 addr v2:172.21.15.177:3300/0 2024-01-21T03:56:30.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02180a8e20 0x7f021813f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 10 monclient(hunting): picked mon.noname-b con 0x7f02180a8e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:30.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f021813fc40 con 0x7f02180a8a40 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02180aa7b0 con 0x7f02180b2210 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0218145700 con 0x7f02180a8e20 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.157+0000 7f021d2a2640 10 monclient(hunting): authenticate will time out at 3536.473445s 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.158+0000 7f02177fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02180a8e20 0x7f021813f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.158+0000 7f0216ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f02180a8a40 0x7f0218145b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56406/0 (socket says 172.21.15.7:56406) 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.158+0000 7f0216ffd640 1 -- 172.21.15.7:0/1660910166 learned_addr learned my addr 172.21.15.7:0/1660910166 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.158+0000 7f02177fe640 10 monclient(hunting): get_auth_request con 0x7f02180a8e20 auth_method 0 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.158+0000 7f02177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.158+0000 7f02177fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.158+0000 7f02177fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.158+0000 7f02167fc640 1 --2- 172.21.15.7:0/1660910166 >> v2:172.21.15.177:3300/0 conn(0x7f02180b2210 0x7f021813f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:30.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.158+0000 7f02167fc640 10 monclient(hunting): get_auth_request con 0x7f02180b2210 auth_method 0 2024-01-21T03:56:30.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.158+0000 7f02167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:30.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.158+0000 7f02167fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:30.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.158+0000 7f02167fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:30.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.159+0000 7f02177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:30.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.159+0000 7f02177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:30.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.159+0000 7f02177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:30.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.159+0000 7f02167fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:30.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.159+0000 7f02167fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:30.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.159+0000 7f02167fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:30.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.160+0000 7f02177fe640 10 monclient(hunting): handle_auth_done global_id 18014 payload 274 2024-01-21T03:56:30.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.160+0000 7f02177fe640 1 -- 172.21.15.7:0/1660910166 >> v2:172.21.15.177:3300/0 conn(0x7f02180b2210 msgr2=0x7f021813f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:30.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.160+0000 7f02177fe640 1 --2- 172.21.15.7:0/1660910166 >> v2:172.21.15.177:3300/0 conn(0x7f02180b2210 0x7f021813f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.160+0000 7f02177fe640 1 -- 172.21.15.7:0/1660910166 >> v1:172.21.15.7:6789/0 conn(0x7f02180a8a40 legacy=0x7f0218145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.160+0000 7f02177fe640 10 monclient: _finish_hunting 0 2024-01-21T03:56:30.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.160+0000 7f02177fe640 1 monclient: found mon.noname-b 2024-01-21T03:56:30.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.160+0000 7f02177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:30.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.160+0000 7f02177fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:56:30.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.160+0000 7f02177fe640 1 -- 172.21.15.7:0/1660910166 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02180aa9b0 con 0x7f02180a8e20 2024-01-21T03:56:30.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.162+0000 7f02177fe640 1 --2- 172.21.15.7:0/1660910166 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02180a8e20 0x7f021813f700 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f02100097a0 tx=0x7f021002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ae95068ab97569c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:30.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f0215ffb640 1 -- 172.21.15.7:0/1660910166 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0210031830 con 0x7f02180a8e20 2024-01-21T03:56:30.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f0215ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:30.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f0215ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:30.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f0215ffb640 10 monclient: dump: 2024-01-21T03:56:30.167 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:30.167 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:30.167 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.167 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.167 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:30.167 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.167 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f0215ffb640 10 monclient: _finish_auth 0 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f0215ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f0215ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f0215ffb640 1 -- 172.21.15.7:0/1660910166 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0210031990 con 0x7f02180a8e20 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f0215ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f0215ffb640 1 -- 172.21.15.7:0/1660910166 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0210031d20 con 0x7f02180a8e20 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f0215ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f0215ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f0215ffb640 10 monclient: dump: 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:30.168 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:30.169 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.169 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.169 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:30.169 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.169 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:30.169 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:30.169 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:30.169 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f021d2a2640 5 monclient: authenticate success, global_id 18014 2024-01-21T03:56:30.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f021d2a2640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:30.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.166+0000 7f021d2a2640 4 set_mon_vals no callback set 2024-01-21T03:56:30.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.167+0000 7f021d2a2640 10 monclient: shutdown 2024-01-21T03:56:30.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.167+0000 7f021d2a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:30.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.167+0000 7f021d2a2640 1 -- 172.21.15.7:0/1660910166 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02180a8e20 msgr2=0x7f021813f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.167+0000 7f021d2a2640 1 --2- 172.21.15.7:0/1660910166 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02180a8e20 0x7f021813f700 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f02100097a0 tx=0x7f021002b8d0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.167+0000 7f021d2a2640 1 -- 172.21.15.7:0/1660910166 shutdown_connections 2024-01-21T03:56:30.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.167+0000 7f021d2a2640 1 --2- 172.21.15.7:0/1660910166 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02180a8e20 0x7f021813f700 unknown :-1 s=CLOSED pgs=3150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.167+0000 7f021d2a2640 1 --2- 172.21.15.7:0/1660910166 >> v2:172.21.15.177:3300/0 conn(0x7f02180b2210 0x7f021813f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.167+0000 7f021d2a2640 1 -- 172.21.15.7:0/1660910166 >> 172.21.15.7:0/1660910166 conn(0x7f0218066f10 msgr2=0x7f02180b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:30.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.167+0000 7f021d2a2640 1 -- 172.21.15.7:0/1660910166 shutdown_connections 2024-01-21T03:56:30.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.167+0000 7f021d2a2640 1 -- 172.21.15.7:0/1660910166 wait complete. 2024-01-21T03:56:30.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.168+0000 7f021d2a2640 10 monclient: build_initial_monmap 2024-01-21T03:56:30.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.168+0000 7f021d2a2640 10 monclient: monmap: 2024-01-21T03:56:30.173 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.168+0000 7f021d2a2640 1 librados: starting msgr at 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.168+0000 7f021d2a2640 1 librados: starting objecter 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.168+0000 7f021d2a2640 1 Processor -- start 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.168+0000 7f021d2a2640 1 -- start start 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.168+0000 7f021d2a2640 1 librados: setting wanted keys 2024-01-21T03:56:30.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.168+0000 7f021d2a2640 1 librados: calling monclient init 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.168+0000 7f021d2a2640 10 monclient: init 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02180a8a40 0x7f0218140e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 10 monclient(hunting): picked mon.noname-b con 0x7f02180a8a40 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02180a8e20 0x7f0218141410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 10 monclient(hunting): picked mon.noname-a con 0x7f02180a8e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02180b2210 0x7f02181dab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 10 monclient(hunting): picked mon.noname-c con 0x7f02180b2210 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0218142f20 con 0x7f02180a8e20 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0218143200 con 0x7f02180a8a40 2024-01-21T03:56:30.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0218141ae0 con 0x7f02180b2210 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f021d2a2640 10 monclient(hunting): authenticate will time out at 3536.485012s 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f02177fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02180b2210 0x7f02181dab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f02167fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02180a8e20 0x7f0218141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f0216ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02180a8a40 0x7f0218140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f02177fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02180b2210 0x7f02181dab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37030/0 (socket says 172.21.15.7:37030) 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f02177fe640 1 -- 172.21.15.7:0/3304894905 learned_addr learned my addr 172.21.15.7:0/3304894905 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f02177fe640 10 monclient(hunting): get_auth_request con 0x7f02180b2210 auth_method 0 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f02177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f02177fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.169+0000 7f02177fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f0216ffd640 10 monclient(hunting): get_auth_request con 0x7f02180a8a40 auth_method 0 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f0216ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f0216ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f0216ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f02167fc640 10 monclient(hunting): get_auth_request con 0x7f02180a8e20 auth_method 0 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f02167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f02167fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f02167fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f02177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f02177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f02177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f0216ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f0216ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f0216ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f02167fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f02167fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:30.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.170+0000 7f02167fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:30.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.173+0000 7f0216ffd640 10 monclient(hunting): handle_auth_done global_id 23161 payload 824 2024-01-21T03:56:30.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.173+0000 7f0216ffd640 1 -- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02180b2210 msgr2=0x7f02181dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.173+0000 7f0216ffd640 1 --2- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f02180b2210 0x7f02181dab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.173+0000 7f0216ffd640 1 -- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02180a8e20 msgr2=0x7f0218141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.173+0000 7f0216ffd640 1 --2- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02180a8e20 0x7f0218141410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.173+0000 7f0216ffd640 10 monclient: _finish_hunting 0 2024-01-21T03:56:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.173+0000 7f0216ffd640 1 monclient: found mon.noname-b 2024-01-21T03:56:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.173+0000 7f0216ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.173+0000 7f0216ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:56:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.173+0000 7f0216ffd640 1 -- 172.21.15.7:0/3304894905 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0210009400 con 0x7f02180a8a40 2024-01-21T03:56:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.175+0000 7f0216ffd640 1 --2- 172.21.15.7:0/3304894905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02180a8a40 0x7f0218140e10 secure :-1 s=READY pgs=5810 cs=0 l=1 rev1=1 crypto rx=0x7f020800ca50 tx=0x7f020800cf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.175+0000 7f020f7fe640 10 client.?.objecter ms_handle_connect 0x7f02180a8a40 2024-01-21T03:56:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.175+0000 7f020f7fe640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f020f7fe640 1 -- 172.21.15.7:0/3304894905 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020800abb0 con 0x7f02180a8a40 2024-01-21T03:56:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f020f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f020f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f020f7fe640 10 monclient: dump: 2024-01-21T03:56:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.180 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f020f7fe640 10 monclient: _finish_auth 0 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f020f7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f020f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f020f7fe640 1 -- 172.21.15.7:0/3304894905 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020800ad10 con 0x7f02180a8a40 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f021d2a2640 5 monclient: authenticate success, global_id 23161 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f021d2a2640 10 monclient: _renew_subs 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f021d2a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f021d2a2640 1 -- 172.21.15.7:0/3304894905 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0218142000 con 0x7f02180a8a40 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f020f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f020f7fe640 1 -- 172.21.15.7:0/3304894905 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0208012620 con 0x7f02180a8a40 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f020f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f020f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f020f7fe640 10 monclient: dump: 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f02157fa640 4 set_mon_vals no callback set 2024-01-21T03:56:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f021d2a2640 10 client.23161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:30.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f021d2a2640 10 monclient: _renew_subs 2024-01-21T03:56:30.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f021d2a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:30.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f021d2a2640 1 -- 172.21.15.7:0/3304894905 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02181db220 con 0x7f02180a8a40 2024-01-21T03:56:30.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.179+0000 7f021d2a2640 1 librados: init done 2024-01-21T03:56:30.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.180+0000 7f021d2a2640 10 client.23161.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:30.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.180+0000 7f021d2a2640 15 client.23161.objecter _session_command_op_assign -1 1 2024-01-21T03:56:30.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.180+0000 7f021d2a2640 20 client.23161.objecter _get_session osd=-1 returning homeless 2024-01-21T03:56:30.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.180+0000 7f021d2a2640 10 client.23161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:30.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.180+0000 7f021d2a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:30.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.180+0000 7f021d2a2640 1 -- 172.21.15.7:0/3304894905 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f02180aad30 con 0x7f02180a8a40 2024-01-21T03:56:30.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.201+0000 7f020f7fe640 1 -- 172.21.15.7:0/3304894905 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f0208007070 con 0x7f02180a8a40 2024-01-21T03:56:30.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.201+0000 7f020f7fe640 1 --2- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f01f806c8b0 0x7f01f806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.201+0000 7f02167fc640 1 --2- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f01f806c8b0 0x7f01f806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:30.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.201+0000 7f02167fc640 10 monclient: get_auth_request con 0x7f01f806c8b0 auth_method 0 2024-01-21T03:56:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.202+0000 7f02167fc640 1 --2- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f01f806c8b0 0x7f01f806ed70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f02180b5870 tx=0x7f0200009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.202+0000 7f020f7fe640 10 client.23161.objecter ms_handle_connect 0x7f01f806c8b0 2024-01-21T03:56:30.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.203+0000 7f020f7fe640 1 -- 172.21.15.7:0/3304894905 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7f0208058660 con 0x7f02180a8a40 2024-01-21T03:56:30.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.203+0000 7f020f7fe640 10 client.23161.objecter ms_dispatch 0x7f0218000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:30.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.203+0000 7f020f7fe640 3 client.23161.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.203+0000 7f020f7fe640 3 client.23161.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f020f7fe640 10 client.23161.objecter checking command 1 2024-01-21T03:56:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f020f7fe640 1 --2- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] conn(0x7f01f80746a0 0x7f01f8076b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f020f7fe640 20 client.23161.objecter _get_session s=0x7f01f8073fd0 osd=1 3 2024-01-21T03:56:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f020f7fe640 20 client.23161.objecter put_session s=0x7f01f8073fd0 osd=1 3 2024-01-21T03:56:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f020f7fe640 15 client.23161.objecter _session_command_op_remove -1 1 2024-01-21T03:56:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f020f7fe640 20 client.23161.objecter _get_session s=0x7f01f8073fd0 osd=1 3 2024-01-21T03:56:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f020f7fe640 20 client.23161.objecter get_session s=0x7f01f8073fd0 osd=1 3 2024-01-21T03:56:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f020f7fe640 15 client.23161.objecter _session_command_op_assign 1 1 2024-01-21T03:56:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f020f7fe640 20 client.23161.objecter put_session s=0x7f01f8073fd0 osd=1 4 2024-01-21T03:56:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f020f7fe640 10 client.23161.objecter _send_command 1 2024-01-21T03:56:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f020f7fe640 1 -- 172.21.15.7:0/3304894905 --> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01f8077210 con 0x7f01f80746a0 2024-01-21T03:56:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f020f7fe640 20 client.23161.objecter dump_active .. 0 homeless 2024-01-21T03:56:30.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f02177fe640 1 --2- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] conn(0x7f01f80746a0 0x7f01f8076b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:30.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.204+0000 7f020f7fe640 1 -- 172.21.15.7:0/3304894905 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=1294) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0208001b60 con 0x7f02180a8a40 2024-01-21T03:56:30.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.205+0000 7f020f7fe640 10 monclient: handle_get_version_reply finishing 1 version 1294 2024-01-21T03:56:30.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.205+0000 7f02177fe640 10 monclient: get_auth_request con 0x7f01f80746a0 auth_method 0 2024-01-21T03:56:30.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.207+0000 7f020f7fe640 10 client.23161.objecter ms_handle_connect 0x7f01f80746a0 2024-01-21T03:56:30.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.207+0000 7f02177fe640 1 --2- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] conn(0x7f01f80746a0 0x7f01f8076b40 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:30.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.233+0000 7f020f7fe640 1 -- 172.21.15.7:0/3304894905 <== osd.1 v2:172.21.15.7:6810/2176696455 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f01f8077210 con 0x7f01f80746a0 2024-01-21T03:56:30.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.233+0000 7f020f7fe640 10 client.23161.objecter ms_dispatch 0x7f0218000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:56:30.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.233+0000 7f020f7fe640 10 client.23161.objecter _finish_command 1 = system:0 2024-01-21T03:56:30.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.233+0000 7f020f7fe640 20 client.23161.objecter put_session s=0x7f01f8073fd0 osd=1 4 2024-01-21T03:56:30.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.233+0000 7f020f7fe640 15 client.23161.objecter _session_command_op_remove 1 1 2024-01-21T03:56:30.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.247+0000 7f021d2a2640 10 client.23161.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T03:56:30.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.247+0000 7f021d2a2640 15 client.23161.objecter _session_command_op_assign -1 2 2024-01-21T03:56:30.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.247+0000 7f021d2a2640 20 client.23161.objecter _get_session s=0x7f01f8073fd0 osd=1 3 2024-01-21T03:56:30.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.247+0000 7f021d2a2640 20 client.23161.objecter put_session s=0x7f01f8073fd0 osd=1 3 2024-01-21T03:56:30.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.247+0000 7f021d2a2640 20 client.23161.objecter _get_session s=0x7f01f8073fd0 osd=1 3 2024-01-21T03:56:30.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.247+0000 7f021d2a2640 15 client.23161.objecter _session_command_op_remove -1 2 2024-01-21T03:56:30.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.247+0000 7f021d2a2640 20 client.23161.objecter get_session s=0x7f01f8073fd0 osd=1 3 2024-01-21T03:56:30.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.247+0000 7f021d2a2640 15 client.23161.objecter _session_command_op_assign 1 2 2024-01-21T03:56:30.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.247+0000 7f021d2a2640 20 client.23161.objecter put_session s=0x7f01f8073fd0 osd=1 4 2024-01-21T03:56:30.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.247+0000 7f021d2a2640 10 client.23161.objecter _send_command 2 2024-01-21T03:56:30.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.247+0000 7f021d2a2640 1 -- 172.21.15.7:0/3304894905 --> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f02180aa1c0 con 0x7f01f80746a0 2024-01-21T03:56:30.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.267+0000 7f020f7fe640 1 -- 172.21.15.7:0/3304894905 <== osd.1 v2:172.21.15.7:6810/2176696455 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f02180aa1c0 con 0x7f01f80746a0 2024-01-21T03:56:30.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.267+0000 7f020f7fe640 10 client.23161.objecter ms_dispatch 0x7f0218000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:56:30.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.267+0000 7f020f7fe640 10 client.23161.objecter _finish_command 2 = system:0 2024-01-21T03:56:30.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.267+0000 7f020f7fe640 20 client.23161.objecter put_session s=0x7f01f8073fd0 osd=1 4 2024-01-21T03:56:30.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.267+0000 7f020f7fe640 15 client.23161.objecter _session_command_op_remove 1 2 2024-01-21T03:56:30.268 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-01-21T03:56:30.269 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T03:56:30.269 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-01-21T03:56:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.267+0000 7f021d2a2640 10 librados: watch_flush enter 2024-01-21T03:56:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.267+0000 7f021d2a2640 10 librados: watch_flush exit 2024-01-21T03:56:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.267+0000 7f021d2a2640 10 client.23161.objecter close_session for osd.1 2024-01-21T03:56:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.267+0000 7f021d2a2640 1 -- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] conn(0x7f01f80746a0 msgr2=0x7f01f8076b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 1 --2- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:6810/2176696455,v1:172.21.15.7:6811/2176696455] conn(0x7f01f80746a0 0x7f01f8076b40 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 20 client.23161.objecter put_session s=0x7f01f8073fd0 osd=1 1 2024-01-21T03:56:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 20 client.23161.objecter shutdown clearing up homeless session... 2024-01-21T03:56:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 10 client.23161.objecter successfully canceled tick 2024-01-21T03:56:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 1 -- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f01f806c8b0 msgr2=0x7f01f806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 1 --2- 172.21.15.7:0/3304894905 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f01f806c8b0 0x7f01f806ed70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f02180b5870 tx=0x7f0200009190 comp rx=0 tx=0).stop 2024-01-21T03:56:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 10 monclient: shutdown 2024-01-21T03:56:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 1 -- 172.21.15.7:0/3304894905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02180a8a40 msgr2=0x7f0218140e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 1 --2- 172.21.15.7:0/3304894905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02180a8a40 0x7f0218140e10 secure :-1 s=READY pgs=5810 cs=0 l=1 rev1=1 crypto rx=0x7f020800ca50 tx=0x7f020800cf10 comp rx=0 tx=0).stop 2024-01-21T03:56:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f02177fe640 1 -- 172.21.15.7:0/3304894905 reap_dead start 2024-01-21T03:56:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 1 -- 172.21.15.7:0/3304894905 shutdown_connections 2024-01-21T03:56:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 1 -- 172.21.15.7:0/3304894905 >> 172.21.15.7:0/3304894905 conn(0x7f0218066f10 msgr2=0x7f02180b6970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 1 -- 172.21.15.7:0/3304894905 shutdown_connections 2024-01-21T03:56:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 1 -- 172.21.15.7:0/3304894905 wait complete. 2024-01-21T03:56:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.268+0000 7f021d2a2640 1 librados: shutdown 2024-01-21T03:56:30.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-01-21T03:56:30.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.395+0000 7fb86bf89640 10 monclient: get_monmap_and_config 2024-01-21T03:56:30.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.395+0000 7fb86bf89640 10 monclient: build_initial_monmap 2024-01-21T03:56:30.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.395+0000 7fb86bf89640 10 monclient: monmap: 2024-01-21T03:56:30.396 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:30.396 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:30.396 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:30.396890+0000 2024-01-21T03:56:30.396 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:30.396890+0000 2024-01-21T03:56:30.397 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:30.397 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.397 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:30.397 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:30.397 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:30.397 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:30.397 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:30.397 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.396+0000 7fb86bf89640 1 Processor -- start 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.396+0000 7fb86bf89640 1 -- start start 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.396+0000 7fb86bf89640 10 monclient: init 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.397+0000 7fb86bf89640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.397+0000 7fb86bf89640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.397+0000 7fb86bf89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb8640a8b20 addr v1:172.21.15.7:6789/0 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.397+0000 7fb86bf89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb8640a9d10 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.397+0000 7fb86bf89640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb8640b2310 0x7fb86413f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.397+0000 7fb86bf89640 10 monclient(hunting): picked mon.noname-a con 0x7fb8640b2310 addr v2:172.21.15.7:3300/0 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.397+0000 7fb86bf89640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8641456f0 con 0x7fb8640a8b20 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.397+0000 7fb86bf89640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8640a98c0 con 0x7fb8640a9d10 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.397+0000 7fb86bf89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.397+0000 7fb86bf89640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb864146290 con 0x7fb8640b2310 2024-01-21T03:56:30.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.397+0000 7fb86bf89640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:30.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.397+0000 7fb86bf89640 10 monclient(hunting): authenticate will time out at 3536.713081s 2024-01-21T03:56:30.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.397+0000 7fb86a526640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb8640b2310 0x7fb86413f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:30.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.398+0000 7fb869524640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fb8640a9d10 0x7fb86413f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:42552/0 (socket says 172.21.15.7:42552) 2024-01-21T03:56:30.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.398+0000 7fb869524640 1 -- 172.21.15.7:0/2866891671 learned_addr learned my addr 172.21.15.7:0/2866891671 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:30.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.398+0000 7fb86a526640 10 monclient(hunting): get_auth_request con 0x7fb8640b2310 auth_method 0 2024-01-21T03:56:30.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.398+0000 7fb86a526640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:30.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.398+0000 7fb86a526640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:30.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.398+0000 7fb86a526640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:30.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.399+0000 7fb86a526640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:30.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.399+0000 7fb86a526640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:30.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.399+0000 7fb86a526640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:30.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.400+0000 7fb868d23640 1 -- 172.21.15.7:0/2866891671 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157012035 0 0) 0x7fb8640a98c0 con 0x7fb8640a9d10 2024-01-21T03:56:30.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.400+0000 7fb868d23640 10 monclient(hunting): _negotiate 2024-01-21T03:56:30.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.400+0000 7fb868d23640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:30.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.400+0000 7fb868d23640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:30.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.400+0000 7fb868d23640 10 monclient(hunting): my global_id is 23164 2024-01-21T03:56:30.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.400+0000 7fb868d23640 1 -- 172.21.15.7:0/2866891671 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb850003520 con 0x7fb8640a9d10 2024-01-21T03:56:30.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.401+0000 7fb86a526640 10 monclient(hunting): handle_auth_done global_id 15852 payload 274 2024-01-21T03:56:30.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.401+0000 7fb86a526640 1 -- 172.21.15.7:0/2866891671 >> v1:172.21.15.177:6789/0 conn(0x7fb8640a9d10 legacy=0x7fb86413f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.401+0000 7fb86a526640 1 -- 172.21.15.7:0/2866891671 >> v1:172.21.15.7:6789/0 conn(0x7fb8640a8b20 legacy=0x7fb864145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.401+0000 7fb86a526640 10 monclient: _finish_hunting 0 2024-01-21T03:56:30.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.401+0000 7fb86a526640 1 monclient: found mon.noname-a 2024-01-21T03:56:30.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.401+0000 7fb86a526640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:30.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.401+0000 7fb86a526640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:56:30.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.401+0000 7fb86a526640 1 -- 172.21.15.7:0/2866891671 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8641463d0 con 0x7fb8640b2310 2024-01-21T03:56:30.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.405+0000 7fb86a526640 1 --2- 172.21.15.7:0/2866891671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640b2310 0x7fb86413f790 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7fb85c009d90 tx=0x7fb85c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994dd8e426800859 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:30.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb868d23640 1 -- 172.21.15.7:0/2866891671 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c02cd00 con 0x7fb8640b2310 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb868d23640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb868d23640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb868d23640 10 monclient: dump: 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb868d23640 10 monclient: _finish_auth 0 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb868d23640 10 monclient: _check_auth_tickets 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb868d23640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb868d23640 1 -- 172.21.15.7:0/2866891671 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c02ce60 con 0x7fb8640b2310 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb86bf89640 5 monclient: authenticate success, global_id 15852 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb86bf89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb868d23640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb86bf89640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb868d23640 1 -- 172.21.15.7:0/2866891671 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c034660 con 0x7fb8640b2310 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb86bf89640 4 set_mon_vals no callback set 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb868d23640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb868d23640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.409+0000 7fb868d23640 10 monclient: dump: 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.410+0000 7fb86bf89640 10 monclient: shutdown 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.410+0000 7fb86bf89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.410+0000 7fb86bf89640 1 -- 172.21.15.7:0/2866891671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640b2310 msgr2=0x7fb86413f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.410+0000 7fb86bf89640 1 --2- 172.21.15.7:0/2866891671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640b2310 0x7fb86413f790 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7fb85c009d90 tx=0x7fb85c02bf50 comp rx=0 tx=0).stop 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.410+0000 7fb86bf89640 1 -- 172.21.15.7:0/2866891671 shutdown_connections 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.410+0000 7fb86bf89640 1 --2- 172.21.15.7:0/2866891671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640b2310 0x7fb86413f790 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.410+0000 7fb86bf89640 1 -- 172.21.15.7:0/2866891671 >> 172.21.15.7:0/2866891671 conn(0x7fb864066fb0 msgr2=0x7fb8640b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:30.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.410+0000 7fb86bf89640 1 -- 172.21.15.7:0/2866891671 shutdown_connections 2024-01-21T03:56:30.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.410+0000 7fb86bf89640 1 -- 172.21.15.7:0/2866891671 wait complete. 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.411+0000 7fb86bf89640 10 monclient: build_initial_monmap 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.411+0000 7fb86bf89640 10 monclient: monmap: 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.411+0000 7fb86bf89640 1 librados: starting msgr at 2024-01-21T03:56:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.411+0000 7fb86bf89640 1 librados: starting objecter 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.411+0000 7fb86bf89640 1 Processor -- start 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.411+0000 7fb86bf89640 1 -- start start 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.411+0000 7fb86bf89640 1 librados: setting wanted keys 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.411+0000 7fb86bf89640 1 librados: calling monclient init 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.411+0000 7fb86bf89640 10 monclient: init 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640a8b20 0x7fb864143020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 10 monclient(hunting): picked mon.noname-a con 0x7fb8640a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640a9d10 0x7fb8641389d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 10 monclient(hunting): picked mon.noname-c con 0x7fb8640a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8640b2310 0x7fb864136b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 10 monclient(hunting): picked mon.noname-b con 0x7fb8640b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8641450b0 con 0x7fb8640a8b20 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864143560 con 0x7fb8640b2310 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8641382d0 con 0x7fb8640a9d10 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb86bf89640 10 monclient(hunting): authenticate will time out at 3536.728119s 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.412+0000 7fb869d25640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640a8b20 0x7fb864143020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb86a526640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8640b2310 0x7fb864136b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb869d25640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640a8b20 0x7fb864143020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44608/0 (socket says 172.21.15.7:44608) 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb869d25640 1 -- 172.21.15.7:0/2594501648 learned_addr learned my addr 172.21.15.7:0/2594501648 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb869d25640 10 monclient(hunting): get_auth_request con 0x7fb8640a8b20 auth_method 0 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb869d25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb869d25640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb869d25640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:30.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb86a526640 10 monclient(hunting): get_auth_request con 0x7fb8640b2310 auth_method 0 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb86a526640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb86a526640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb86a526640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb869d25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb869d25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb869d25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb86a526640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb86a526640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.413+0000 7fb86a526640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.416+0000 7fb86a526640 10 monclient(hunting): handle_auth_done global_id 23167 payload 824 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.416+0000 7fb86a526640 1 -- 172.21.15.7:0/2594501648 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640a9d10 msgr2=0x7fb8641389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.416+0000 7fb86a526640 1 --2- 172.21.15.7:0/2594501648 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640a9d10 0x7fb8641389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.416+0000 7fb86a526640 1 -- 172.21.15.7:0/2594501648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640a8b20 msgr2=0x7fb864143020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.416+0000 7fb86a526640 1 --2- 172.21.15.7:0/2594501648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640a8b20 0x7fb864143020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.416+0000 7fb86a526640 10 monclient: _finish_hunting 0 2024-01-21T03:56:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.416+0000 7fb86a526640 1 monclient: found mon.noname-b 2024-01-21T03:56:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.416+0000 7fb86a526640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.416+0000 7fb86a526640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:56:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.416+0000 7fb86a526640 1 -- 172.21.15.7:0/2594501648 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c009a70 con 0x7fb8640b2310 2024-01-21T03:56:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.416+0000 7fb869d25640 1 --2- 172.21.15.7:0/2594501648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640a8b20 0x7fb864143020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T03:56:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.418+0000 7fb86a526640 1 --2- 172.21.15.7:0/2594501648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8640b2310 0x7fb864136b90 secure :-1 s=READY pgs=5812 cs=0 l=1 rev1=1 crypto rx=0x7fb85c033000 tx=0x7fb85c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.418+0000 7fb85a7fc640 10 client.?.objecter ms_handle_connect 0x7fb8640b2310 2024-01-21T03:56:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.418+0000 7fb85a7fc640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:30.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85a7fc640 1 -- 172.21.15.7:0/2594501648 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c03d600 con 0x7fb8640b2310 2024-01-21T03:56:30.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:30.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:30.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85a7fc640 10 monclient: dump: 2024-01-21T03:56:30.423 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:30.423 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:30.423 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.423 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.423 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:30.423 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.423 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:30.423 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:30.423 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85a7fc640 10 monclient: _finish_auth 0 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85a7fc640 10 monclient: _check_auth_tickets 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85a7fc640 1 -- 172.21.15.7:0/2594501648 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c03d780 con 0x7fb8640b2310 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb86bf89640 5 monclient: authenticate success, global_id 23167 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb86bf89640 10 monclient: _renew_subs 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb86bf89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb86bf89640 1 -- 172.21.15.7:0/2594501648 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8641385b0 con 0x7fb8640b2310 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85a7fc640 1 -- 172.21.15.7:0/2594501648 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c03db30 con 0x7fb8640b2310 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85a7fc640 10 monclient: dump: 2024-01-21T03:56:30.424 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb85bfff640 4 set_mon_vals no callback set 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb86bf89640 10 client.23167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb86bf89640 10 monclient: _renew_subs 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb86bf89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb86bf89640 1 -- 172.21.15.7:0/2594501648 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb864138f10 con 0x7fb8640b2310 2024-01-21T03:56:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.422+0000 7fb86bf89640 1 librados: init done 2024-01-21T03:56:30.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.423+0000 7fb86bf89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:30.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.423+0000 7fb86bf89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:30.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.423+0000 7fb86bf89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:30.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.423+0000 7fb86bf89640 1 -- 172.21.15.7:0/2594501648 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8641390a0 con 0x7fb8640b2310 2024-01-21T03:56:30.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.441+0000 7fb85a7fc640 1 -- 172.21.15.7:0/2594501648 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb85c03b040 con 0x7fb8640b2310 2024-01-21T03:56:30.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.442+0000 7fb85a7fc640 1 --2- 172.21.15.7:0/2594501648 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb84806c8b0 0x7fb84806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.442+0000 7fb869d25640 1 --2- 172.21.15.7:0/2594501648 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb84806c8b0 0x7fb84806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.442+0000 7fb869d25640 10 monclient: get_auth_request con 0x7fb84806c8b0 auth_method 0 2024-01-21T03:56:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.442+0000 7fb869d25640 1 --2- 172.21.15.7:0/2594501648 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb84806c8b0 0x7fb84806ed70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb854001c30 tx=0x7fb854009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.442+0000 7fb85a7fc640 10 client.23167.objecter ms_handle_connect 0x7fb84806c8b0 2024-01-21T03:56:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.444+0000 7fb85a7fc640 1 -- 172.21.15.7:0/2594501648 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7fb85c07ad40 con 0x7fb8640b2310 2024-01-21T03:56:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.444+0000 7fb85a7fc640 10 client.23167.objecter ms_dispatch 0x7fb864000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.444+0000 7fb85a7fc640 3 client.23167.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.444+0000 7fb85a7fc640 3 client.23167.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.444+0000 7fb85a7fc640 20 client.23167.objecter dump_active .. 0 homeless 2024-01-21T03:56:30.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.672+0000 7fb85a7fc640 1 -- 172.21.15.7:0/2594501648 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb85c077080 con 0x7fb8640b2310 2024-01-21T03:56:30.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.672+0000 7fb85a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:30.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.672+0000 7fb85a7fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.853+0000 7fb86bf89640 1 -- 172.21.15.7:0/2594501648 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8640a9150 con 0x7fb84806c8b0 2024-01-21T03:56:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.854+0000 7fb85a7fc640 1 -- 172.21.15.7:0/2594501648 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb8540018d0 con 0x7fb84806c8b0 2024-01-21T03:56:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 1 to deep-scrub 2024-01-21T03:56:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.855+0000 7fb86bf89640 10 librados: watch_flush enter 2024-01-21T03:56:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.855+0000 7fb86bf89640 10 librados: watch_flush exit 2024-01-21T03:56:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.855+0000 7fb86bf89640 20 client.23167.objecter shutdown clearing up homeless session... 2024-01-21T03:56:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.855+0000 7fb86bf89640 10 client.23167.objecter successfully canceled tick 2024-01-21T03:56:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 1 -- 172.21.15.7:0/2594501648 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb84806c8b0 msgr2=0x7fb84806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 1 --2- 172.21.15.7:0/2594501648 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb84806c8b0 0x7fb84806ed70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb854001c30 tx=0x7fb854009040 comp rx=0 tx=0).stop 2024-01-21T03:56:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 10 monclient: shutdown 2024-01-21T03:56:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 1 -- 172.21.15.7:0/2594501648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8640b2310 msgr2=0x7fb864136b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 1 --2- 172.21.15.7:0/2594501648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8640b2310 0x7fb864136b90 secure :-1 s=READY pgs=5812 cs=0 l=1 rev1=1 crypto rx=0x7fb85c033000 tx=0x7fb85c004a40 comp rx=0 tx=0).stop 2024-01-21T03:56:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 1 -- 172.21.15.7:0/2594501648 shutdown_connections 2024-01-21T03:56:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 1 --2- 172.21.15.7:0/2594501648 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb84806c8b0 0x7fb84806ed70 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 1 --2- 172.21.15.7:0/2594501648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8640b2310 0x7fb864136b90 unknown :-1 s=CLOSED pgs=5812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 1 --2- 172.21.15.7:0/2594501648 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640a9d10 0x7fb8641389d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 1 --2- 172.21.15.7:0/2594501648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640a8b20 0x7fb864143020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 1 -- 172.21.15.7:0/2594501648 >> 172.21.15.7:0/2594501648 conn(0x7fb864066fb0 msgr2=0x7fb8640b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 1 -- 172.21.15.7:0/2594501648 shutdown_connections 2024-01-21T03:56:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 1 -- 172.21.15.7:0/2594501648 wait complete. 2024-01-21T03:56:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.856+0000 7fb86bf89640 1 librados: shutdown 2024-01-21T03:56:30.868 INFO:tasks.ceph:Scrubbing osd.2 2024-01-21T03:56:30.868 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.978+0000 7fc2dff2c640 10 monclient: get_monmap_and_config 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.978+0000 7fc2dff2c640 10 monclient: build_initial_monmap 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.978+0000 7fc2dff2c640 10 monclient: monmap: 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:30.979431+0000 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:30.979431+0000 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:30.979 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:30.980 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 1 Processor -- start 2024-01-21T03:56:30.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 1 -- start start 2024-01-21T03:56:30.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 10 monclient: init 2024-01-21T03:56:30.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:30.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc2d80a8ab0 0x7fc2d81426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 10 monclient(hunting): picked mon.noname-a con 0x7fc2d80a8ab0 addr v2:172.21.15.7:3300/0 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2d8142bf0 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2d813f950 0x7fc2d814c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 10 monclient(hunting): picked mon.noname-b con 0x7fc2d813f950 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d80aaa20 con 0x7fc2d8142bf0 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d80aa820 con 0x7fc2d80a8ab0 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d80aad00 con 0x7fc2d813f950 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.979+0000 7fc2dff2c640 10 monclient(hunting): authenticate will time out at 3537.295636s 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.980+0000 7fc2de4c9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2d813f950 0x7fc2d814c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.980+0000 7fc2ddcc8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc2d80a8ab0 0x7fc2d81426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:30.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.980+0000 7fc2de4c9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2d813f950 0x7fc2d814c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37044/0 (socket says 172.21.15.7:37044) 2024-01-21T03:56:30.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.980+0000 7fc2de4c9640 1 -- 172.21.15.7:0/566795353 learned_addr learned my addr 172.21.15.7:0/566795353 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:30.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.980+0000 7fc2de4c9640 10 monclient(hunting): get_auth_request con 0x7fc2d813f950 auth_method 0 2024-01-21T03:56:30.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.980+0000 7fc2de4c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:30.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.980+0000 7fc2de4c9640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:30.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.980+0000 7fc2de4c9640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:30.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.981+0000 7fc2ddcc8640 10 monclient(hunting): get_auth_request con 0x7fc2d80a8ab0 auth_method 0 2024-01-21T03:56:30.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.981+0000 7fc2ddcc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:30.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.981+0000 7fc2ddcc8640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:30.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.981+0000 7fc2ddcc8640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:30.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.981+0000 7fc2de4c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:30.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.981+0000 7fc2de4c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:30.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.981+0000 7fc2de4c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:30.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.981+0000 7fc2ddcc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:30.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.981+0000 7fc2ddcc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:30.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.981+0000 7fc2ddcc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:30.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.983+0000 7fc2de4c9640 10 monclient(hunting): handle_auth_done global_id 18020 payload 274 2024-01-21T03:56:30.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.983+0000 7fc2de4c9640 1 -- 172.21.15.7:0/566795353 >> v2:172.21.15.7:3300/0 conn(0x7fc2d80a8ab0 msgr2=0x7fc2d81426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:30.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.983+0000 7fc2de4c9640 1 --2- 172.21.15.7:0/566795353 >> v2:172.21.15.7:3300/0 conn(0x7fc2d80a8ab0 0x7fc2d81426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.983+0000 7fc2de4c9640 1 -- 172.21.15.7:0/566795353 >> v1:172.21.15.177:6789/0 conn(0x7fc2d8142bf0 legacy=0x7fc2d813f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.983+0000 7fc2de4c9640 10 monclient: _finish_hunting 0 2024-01-21T03:56:30.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.983+0000 7fc2de4c9640 1 monclient: found mon.noname-b 2024-01-21T03:56:30.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.983+0000 7fc2de4c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:30.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.983+0000 7fc2de4c9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:56:30.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.983+0000 7fc2de4c9640 1 -- 172.21.15.7:0/566795353 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d80aafe0 con 0x7fc2d813f950 2024-01-21T03:56:30.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.985+0000 7fc2de4c9640 1 --2- 172.21.15.7:0/566795353 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2d813f950 0x7fc2d814c900 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009e50 tx=0x7fc2d002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b3db4b79354eda server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dccc6640 1 -- 172.21.15.7:0/566795353 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d002ca70 con 0x7fc2d813f950 2024-01-21T03:56:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dccc6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dccc6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dccc6640 10 monclient: dump: 2024-01-21T03:56:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dccc6640 10 monclient: _finish_auth 0 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dccc6640 10 monclient: _check_auth_tickets 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dccc6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dccc6640 1 -- 172.21.15.7:0/566795353 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d002cbd0 con 0x7fc2d813f950 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dff2c640 5 monclient: authenticate success, global_id 18020 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dff2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dccc6640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dff2c640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dccc6640 1 -- 172.21.15.7:0/566795353 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00343f0 con 0x7fc2d813f950 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dff2c640 4 set_mon_vals no callback set 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dccc6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dccc6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.993+0000 7fc2dccc6640 10 monclient: dump: 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.994+0000 7fc2dff2c640 10 monclient: shutdown 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.994+0000 7fc2dff2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.994+0000 7fc2dff2c640 1 -- 172.21.15.7:0/566795353 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2d813f950 msgr2=0x7fc2d814c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.994+0000 7fc2dff2c640 1 --2- 172.21.15.7:0/566795353 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2d813f950 0x7fc2d814c900 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009e50 tx=0x7fc2d002bed0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.994+0000 7fc2dff2c640 1 -- 172.21.15.7:0/566795353 shutdown_connections 2024-01-21T03:56:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.994+0000 7fc2dff2c640 1 --2- 172.21.15.7:0/566795353 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2d813f950 0x7fc2d814c900 unknown :-1 s=CLOSED pgs=3151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.994+0000 7fc2dff2c640 1 --2- 172.21.15.7:0/566795353 >> v2:172.21.15.7:3300/0 conn(0x7fc2d80a8ab0 0x7fc2d81426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.994+0000 7fc2dff2c640 1 -- 172.21.15.7:0/566795353 >> 172.21.15.7:0/566795353 conn(0x7fc2d8066f00 msgr2=0x7fc2d80b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.994+0000 7fc2dff2c640 1 -- 172.21.15.7:0/566795353 shutdown_connections 2024-01-21T03:56:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.994+0000 7fc2dff2c640 1 -- 172.21.15.7:0/566795353 wait complete. 2024-01-21T03:56:31.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.995+0000 7fc2dff2c640 10 monclient: build_initial_monmap 2024-01-21T03:56:31.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.995+0000 7fc2dff2c640 10 monclient: monmap: 2024-01-21T03:56:31.001 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:31.001 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:31.001 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:31.001 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.995+0000 7fc2dff2c640 1 librados: starting msgr at 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.995+0000 7fc2dff2c640 1 librados: starting objecter 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 1 Processor -- start 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 1 -- start start 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 1 librados: setting wanted keys 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 1 librados: calling monclient init 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 10 monclient: init 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:56:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2d80a8ab0 0x7fc2d80aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 10 monclient(hunting): picked mon.noname-b con 0x7fc2d80a8ab0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2d813f950 0x7fc2d80adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 10 monclient(hunting): picked mon.noname-c con 0x7fc2d813f950 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2d81d7510 0x7fc2d81d79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 10 monclient(hunting): picked mon.noname-a con 0x7fc2d81d7510 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d80af390 con 0x7fc2d81d7510 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d80b07d0 con 0x7fc2d80a8ab0 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d80b00d0 con 0x7fc2d813f950 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.996+0000 7fc2dff2c640 10 monclient(hunting): authenticate will time out at 3537.312421s 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.997+0000 7fc2dd4c7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2d813f950 0x7fc2d80adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.997+0000 7fc2ddcc8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2d80a8ab0 0x7fc2d80aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.997+0000 7fc2dd4c7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2d813f950 0x7fc2d80adfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37058/0 (socket says 172.21.15.7:37058) 2024-01-21T03:56:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.997+0000 7fc2dd4c7640 1 -- 172.21.15.7:0/3779463755 learned_addr learned my addr 172.21.15.7:0/3779463755 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.997+0000 7fc2dd4c7640 10 monclient(hunting): get_auth_request con 0x7fc2d813f950 auth_method 0 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.997+0000 7fc2dd4c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.997+0000 7fc2dd4c7640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.997+0000 7fc2dd4c7640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.997+0000 7fc2ddcc8640 10 monclient(hunting): get_auth_request con 0x7fc2d80a8ab0 auth_method 0 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.997+0000 7fc2ddcc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.997+0000 7fc2ddcc8640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.997+0000 7fc2ddcc8640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.998+0000 7fc2dd4c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.998+0000 7fc2dd4c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.998+0000 7fc2dd4c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.998+0000 7fc2ddcc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.998+0000 7fc2ddcc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.998+0000 7fc2ddcc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.998+0000 7fc2de4c9640 1 --2- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2d81d7510 0x7fc2d81d79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.998+0000 7fc2de4c9640 10 monclient(hunting): get_auth_request con 0x7fc2d81d7510 auth_method 0 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.998+0000 7fc2de4c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.998+0000 7fc2de4c9640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.998+0000 7fc2de4c9640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.999+0000 7fc2de4c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.999+0000 7fc2de4c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:30.999+0000 7fc2de4c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.000+0000 7fc2ddcc8640 10 monclient(hunting): handle_auth_done global_id 23173 payload 824 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.000+0000 7fc2ddcc8640 1 -- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2d813f950 msgr2=0x7fc2d80adfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.000+0000 7fc2ddcc8640 1 --2- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc2d813f950 0x7fc2d80adfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.000+0000 7fc2ddcc8640 1 -- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2d81d7510 msgr2=0x7fc2d81d79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.000+0000 7fc2ddcc8640 1 --2- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2d81d7510 0x7fc2d81d79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.000+0000 7fc2ddcc8640 10 monclient: _finish_hunting 0 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.000+0000 7fc2ddcc8640 1 monclient: found mon.noname-b 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.000+0000 7fc2ddcc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.000+0000 7fc2ddcc8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:56:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.000+0000 7fc2ddcc8640 1 -- 172.21.15.7:0/3779463755 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d0009b30 con 0x7fc2d80a8ab0 2024-01-21T03:56:31.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.002+0000 7fc2ddcc8640 1 --2- 172.21.15.7:0/3779463755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2d80a8ab0 0x7fc2d80aece0 secure :-1 s=READY pgs=5814 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400e850 tx=0x7fc2d400ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:31.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.002+0000 7fc2ce7fc640 10 client.?.objecter ms_handle_connect 0x7fc2d80a8ab0 2024-01-21T03:56:31.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.002+0000 7fc2ce7fc640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:31.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2ce7fc640 1 -- 172.21.15.7:0/3779463755 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d40095b0 con 0x7fc2d80a8ab0 2024-01-21T03:56:31.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:31.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2ce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:31.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2ce7fc640 10 monclient: dump: 2024-01-21T03:56:31.007 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2ce7fc640 10 monclient: _finish_auth 0 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2ce7fc640 10 monclient: _check_auth_tickets 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2ce7fc640 1 -- 172.21.15.7:0/3779463755 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d400c940 con 0x7fc2d80a8ab0 2024-01-21T03:56:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2ce7fc640 1 -- 172.21.15.7:0/3779463755 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d400ccd0 con 0x7fc2d80a8ab0 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2ce7fc640 10 monclient: dump: 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2dff2c640 5 monclient: authenticate success, global_id 23173 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2dff2c640 10 monclient: _renew_subs 2024-01-21T03:56:31.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2dff2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2dff2c640 1 -- 172.21.15.7:0/3779463755 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d80b03b0 con 0x7fc2d80a8ab0 2024-01-21T03:56:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2cffff640 4 set_mon_vals no callback set 2024-01-21T03:56:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2dff2c640 10 client.23173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2dff2c640 10 monclient: _renew_subs 2024-01-21T03:56:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2dff2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2dff2c640 1 -- 172.21.15.7:0/3779463755 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d81d80c0 con 0x7fc2d80a8ab0 2024-01-21T03:56:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.006+0000 7fc2dff2c640 1 librados: init done 2024-01-21T03:56:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.007+0000 7fc2dff2c640 10 client.23173.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.007+0000 7fc2dff2c640 15 client.23173.objecter _session_command_op_assign -1 1 2024-01-21T03:56:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.007+0000 7fc2dff2c640 20 client.23173.objecter _get_session osd=-1 returning homeless 2024-01-21T03:56:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.007+0000 7fc2dff2c640 10 client.23173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.007+0000 7fc2dff2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:31.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.007+0000 7fc2dff2c640 1 -- 172.21.15.7:0/3779463755 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc2d804e970 con 0x7fc2d80a8ab0 2024-01-21T03:56:31.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.027+0000 7fc2ce7fc640 1 -- 172.21.15.7:0/3779463755 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc2d4026480 con 0x7fc2d80a8ab0 2024-01-21T03:56:31.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.027+0000 7fc2ce7fc640 1 --2- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fc2bc06c8e0 0x7fc2bc06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.027+0000 7fc2dd4c7640 1 --2- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fc2bc06c8e0 0x7fc2bc06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.027+0000 7fc2dd4c7640 10 monclient: get_auth_request con 0x7fc2bc06c8e0 auth_method 0 2024-01-21T03:56:31.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.027+0000 7fc2ce7fc640 10 client.23173.objecter ms_handle_connect 0x7fc2bc06c8e0 2024-01-21T03:56:31.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.027+0000 7fc2dd4c7640 1 --2- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fc2bc06c8e0 0x7fc2bc06eda0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80aced0 tx=0x7fc2c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:31.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 1 -- 172.21.15.7:0/3779463755 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7fc2d400ce30 con 0x7fc2d80a8ab0 2024-01-21T03:56:31.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 10 client.23173.objecter ms_dispatch 0x7fc2d8000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:31.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 3 client.23173.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 3 client.23173.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 10 client.23173.objecter checking command 1 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 1 --2- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] conn(0x7fc2bc0746d0 0x7fc2bc076b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 20 client.23173.objecter _get_session s=0x7fc2bc074000 osd=2 3 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 20 client.23173.objecter put_session s=0x7fc2bc074000 osd=2 3 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 15 client.23173.objecter _session_command_op_remove -1 1 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 20 client.23173.objecter _get_session s=0x7fc2bc074000 osd=2 3 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 20 client.23173.objecter get_session s=0x7fc2bc074000 osd=2 3 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 15 client.23173.objecter _session_command_op_assign 2 1 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 20 client.23173.objecter put_session s=0x7fc2bc074000 osd=2 4 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 10 client.23173.objecter _send_command 1 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 1 -- 172.21.15.7:0/3779463755 --> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc2bc077240 con 0x7fc2bc0746d0 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 20 client.23173.objecter dump_active .. 0 homeless 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 1 -- 172.21.15.7:0/3779463755 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=1294) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2d4058840 con 0x7fc2d80a8ab0 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.028+0000 7fc2ce7fc640 10 monclient: handle_get_version_reply finishing 1 version 1294 2024-01-21T03:56:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.029+0000 7fc2de4c9640 1 --2- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] conn(0x7fc2bc0746d0 0x7fc2bc076b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.029+0000 7fc2de4c9640 10 monclient: get_auth_request con 0x7fc2bc0746d0 auth_method 0 2024-01-21T03:56:31.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.032+0000 7fc2de4c9640 1 --2- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] conn(0x7fc2bc0746d0 0x7fc2bc076b70 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:31.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.032+0000 7fc2ce7fc640 10 client.23173.objecter ms_handle_connect 0x7fc2bc0746d0 2024-01-21T03:56:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.057+0000 7fc2ce7fc640 1 -- 172.21.15.7:0/3779463755 <== osd.2 v2:172.21.15.7:6826/888884750 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc2bc077240 con 0x7fc2bc0746d0 2024-01-21T03:56:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.057+0000 7fc2ce7fc640 10 client.23173.objecter ms_dispatch 0x7fc2d8000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:56:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.057+0000 7fc2ce7fc640 10 client.23173.objecter _finish_command 1 = system:0 2024-01-21T03:56:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.057+0000 7fc2ce7fc640 20 client.23173.objecter put_session s=0x7fc2bc074000 osd=2 4 2024-01-21T03:56:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.057+0000 7fc2ce7fc640 15 client.23173.objecter _session_command_op_remove 2 1 2024-01-21T03:56:31.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.071+0000 7fc2dff2c640 10 client.23173.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T03:56:31.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.071+0000 7fc2dff2c640 15 client.23173.objecter _session_command_op_assign -1 2 2024-01-21T03:56:31.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.071+0000 7fc2dff2c640 20 client.23173.objecter _get_session s=0x7fc2bc074000 osd=2 3 2024-01-21T03:56:31.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.071+0000 7fc2dff2c640 20 client.23173.objecter put_session s=0x7fc2bc074000 osd=2 3 2024-01-21T03:56:31.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.071+0000 7fc2dff2c640 20 client.23173.objecter _get_session s=0x7fc2bc074000 osd=2 3 2024-01-21T03:56:31.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.071+0000 7fc2dff2c640 15 client.23173.objecter _session_command_op_remove -1 2 2024-01-21T03:56:31.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.071+0000 7fc2dff2c640 20 client.23173.objecter get_session s=0x7fc2bc074000 osd=2 3 2024-01-21T03:56:31.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.071+0000 7fc2dff2c640 15 client.23173.objecter _session_command_op_assign 2 2 2024-01-21T03:56:31.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.071+0000 7fc2dff2c640 20 client.23173.objecter put_session s=0x7fc2bc074000 osd=2 4 2024-01-21T03:56:31.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.071+0000 7fc2dff2c640 10 client.23173.objecter _send_command 2 2024-01-21T03:56:31.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.071+0000 7fc2dff2c640 1 -- 172.21.15.7:0/3779463755 --> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc2d80b0240 con 0x7fc2bc0746d0 2024-01-21T03:56:31.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2ce7fc640 1 -- 172.21.15.7:0/3779463755 <== osd.2 v2:172.21.15.7:6826/888884750 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fc2d80b0240 con 0x7fc2bc0746d0 2024-01-21T03:56:31.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2ce7fc640 10 client.23173.objecter ms_dispatch 0x7fc2d8000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:56:31.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2ce7fc640 10 client.23173.objecter _finish_command 2 = system:0 2024-01-21T03:56:31.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2ce7fc640 20 client.23173.objecter put_session s=0x7fc2bc074000 osd=2 4 2024-01-21T03:56:31.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2ce7fc640 15 client.23173.objecter _session_command_op_remove 2 2 2024-01-21T03:56:31.085 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-01-21T03:56:31.085 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T03:56:31.085 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 10 librados: watch_flush enter 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 10 librados: watch_flush exit 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 10 client.23173.objecter close_session for osd.2 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 1 -- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] conn(0x7fc2bc0746d0 msgr2=0x7fc2bc076b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 1 --2- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:6826/888884750,v1:172.21.15.7:6827/888884750] conn(0x7fc2bc0746d0 0x7fc2bc076b70 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 20 client.23173.objecter put_session s=0x7fc2bc074000 osd=2 1 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 20 client.23173.objecter shutdown clearing up homeless session... 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 10 client.23173.objecter successfully canceled tick 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 1 -- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fc2bc06c8e0 msgr2=0x7fc2bc06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 1 --2- 172.21.15.7:0/3779463755 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fc2bc06c8e0 0x7fc2bc06eda0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80aced0 tx=0x7fc2c8009040 comp rx=0 tx=0).stop 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 10 monclient: shutdown 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 1 -- 172.21.15.7:0/3779463755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2d80a8ab0 msgr2=0x7fc2d80aece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.084+0000 7fc2dff2c640 1 --2- 172.21.15.7:0/3779463755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2d80a8ab0 0x7fc2d80aece0 secure :-1 s=READY pgs=5814 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400e850 tx=0x7fc2d400ed10 comp rx=0 tx=0).stop 2024-01-21T03:56:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.085+0000 7fc2de4c9640 1 -- 172.21.15.7:0/3779463755 reap_dead start 2024-01-21T03:56:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.085+0000 7fc2dff2c640 1 -- 172.21.15.7:0/3779463755 shutdown_connections 2024-01-21T03:56:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.085+0000 7fc2dff2c640 1 -- 172.21.15.7:0/3779463755 >> 172.21.15.7:0/3779463755 conn(0x7fc2d8066f00 msgr2=0x7fc2d813e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.085+0000 7fc2dff2c640 1 -- 172.21.15.7:0/3779463755 shutdown_connections 2024-01-21T03:56:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.085+0000 7fc2dff2c640 1 -- 172.21.15.7:0/3779463755 wait complete. 2024-01-21T03:56:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.085+0000 7fc2dff2c640 1 librados: shutdown 2024-01-21T03:56:31.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-01-21T03:56:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.199+0000 7f5427a2b640 10 monclient: get_monmap_and_config 2024-01-21T03:56:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.199+0000 7f5427a2b640 10 monclient: build_initial_monmap 2024-01-21T03:56:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.199+0000 7f5427a2b640 10 monclient: monmap: 2024-01-21T03:56:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:31.200707+0000 2024-01-21T03:56:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:31.200707+0000 2024-01-21T03:56:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:31.200 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:31.201 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:31.201 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:31.201 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.200+0000 7f5427a2b640 1 Processor -- start 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.200+0000 7f5427a2b640 1 -- start start 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.200+0000 7f5427a2b640 10 monclient: init 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f54200a8ab0 0x7f542013efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 10 monclient(hunting): picked mon.noname-a con 0x7f54200a8ab0 addr v2:172.21.15.7:3300/0 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f542013f500 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54201429f0 0x7f542014c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 10 monclient(hunting): picked mon.noname-b con 0x7f54201429f0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542013f930 con 0x7f542013f500 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54200aaa20 con 0x7f54200a8ab0 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:31.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54200aad00 con 0x7f54201429f0 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5427a2b640 10 monclient(hunting): authenticate will time out at 3537.516851s 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5425fc8640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54201429f0 0x7f542014c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f54257c7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f54200a8ab0 0x7f542013efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5424fc6640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f542013f500 0x7f54201422e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:42574/0 (socket says 172.21.15.7:42574) 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5424fc6640 1 -- 172.21.15.7:0/1981741367 learned_addr learned my addr 172.21.15.7:0/1981741367 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5425fc8640 10 monclient(hunting): get_auth_request con 0x7f54201429f0 auth_method 0 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5425fc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5425fc8640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.201+0000 7f5425fc8640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.202+0000 7f54257c7640 10 monclient(hunting): get_auth_request con 0x7f54200a8ab0 auth_method 0 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.202+0000 7f54257c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.202+0000 7f54257c7640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.202+0000 7f54257c7640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.202+0000 7f5425fc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.202+0000 7f5425fc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:31.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.202+0000 7f5425fc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:31.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.202+0000 7f54257c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:31.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.202+0000 7f54257c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:31.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.202+0000 7f54257c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5417fff640 1 -- 172.21.15.7:0/1981741367 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842138012 0 0) 0x7f542013f930 con 0x7f542013f500 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5417fff640 10 monclient(hunting): _negotiate 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5417fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5417fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5417fff640 10 monclient(hunting): my global_id is 23176 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5417fff640 1 -- 172.21.15.7:0/1981741367 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f540c003520 con 0x7f542013f500 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5425fc8640 10 monclient(hunting): handle_auth_done global_id 18026 payload 274 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5425fc8640 1 -- 172.21.15.7:0/1981741367 >> v2:172.21.15.7:3300/0 conn(0x7f54200a8ab0 msgr2=0x7f542013efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5425fc8640 1 --2- 172.21.15.7:0/1981741367 >> v2:172.21.15.7:3300/0 conn(0x7f54200a8ab0 0x7f542013efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5425fc8640 1 -- 172.21.15.7:0/1981741367 >> v1:172.21.15.177:6789/0 conn(0x7f542013f500 legacy=0x7f54201422e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5425fc8640 10 monclient: _finish_hunting 0 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5425fc8640 1 monclient: found mon.noname-b 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5425fc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:31.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5425fc8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:56:31.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.204+0000 7f5425fc8640 1 -- 172.21.15.7:0/1981741367 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54200aafe0 con 0x7f54201429f0 2024-01-21T03:56:31.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.206+0000 7f5425fc8640 1 --2- 172.21.15.7:0/1981741367 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54201429f0 0x7f542014c950 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7f5418009d90 tx=0x7f541802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=154486ba36c0b8e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:31.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.208+0000 7f5417fff640 1 -- 172.21.15.7:0/1981741367 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541802cc50 con 0x7f54201429f0 2024-01-21T03:56:31.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.208+0000 7f5417fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:31.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.208+0000 7f5417fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.208+0000 7f5417fff640 10 monclient: dump: 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.208+0000 7f5417fff640 10 monclient: _finish_auth 0 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.208+0000 7f5417fff640 10 monclient: _check_auth_tickets 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.208+0000 7f5417fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:31.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.208+0000 7f5427a2b640 5 monclient: authenticate success, global_id 18026 2024-01-21T03:56:31.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.208+0000 7f5427a2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:31.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5417fff640 1 -- 172.21.15.7:0/1981741367 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541802cdb0 con 0x7f54201429f0 2024-01-21T03:56:31.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5417fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:31.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5417fff640 1 -- 172.21.15.7:0/1981741367 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418034640 con 0x7f54201429f0 2024-01-21T03:56:31.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5427a2b640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:31.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5427a2b640 4 set_mon_vals no callback set 2024-01-21T03:56:31.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5417fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:31.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5417fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:31.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5417fff640 10 monclient: dump: 2024-01-21T03:56:31.211 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:31.211 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:31.211 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.211 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.211 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:31.211 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.211 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:31.211 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:31.211 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:31.211 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5427a2b640 10 monclient: shutdown 2024-01-21T03:56:31.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5427a2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5427a2b640 1 -- 172.21.15.7:0/1981741367 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54201429f0 msgr2=0x7f542014c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5427a2b640 1 --2- 172.21.15.7:0/1981741367 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54201429f0 0x7f542014c950 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7f5418009d90 tx=0x7f541802bea0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5427a2b640 1 -- 172.21.15.7:0/1981741367 shutdown_connections 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5427a2b640 1 --2- 172.21.15.7:0/1981741367 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54201429f0 0x7f542014c950 unknown :-1 s=CLOSED pgs=3152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5427a2b640 1 --2- 172.21.15.7:0/1981741367 >> v2:172.21.15.7:3300/0 conn(0x7f54200a8ab0 0x7f542013efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5427a2b640 1 -- 172.21.15.7:0/1981741367 >> 172.21.15.7:0/1981741367 conn(0x7f5420066f50 msgr2=0x7f54200b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5427a2b640 1 -- 172.21.15.7:0/1981741367 shutdown_connections 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.210+0000 7f5427a2b640 1 -- 172.21.15.7:0/1981741367 wait complete. 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.211+0000 7f5427a2b640 10 monclient: build_initial_monmap 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.211+0000 7f5427a2b640 10 monclient: monmap: 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.215 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.211+0000 7f5427a2b640 1 librados: starting msgr at 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.211+0000 7f5427a2b640 1 librados: starting objecter 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 1 Processor -- start 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 1 -- start start 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 1 librados: setting wanted keys 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 1 librados: calling monclient init 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 10 monclient: init 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54200a8ab0 0x7f54201da610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 10 monclient(hunting): picked mon.noname-b con 0x7f54200a8ab0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f542013f500 0x7f54201dab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 10 monclient(hunting): picked mon.noname-c con 0x7f542013f500 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:31.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54201429f0 0x7f54201d7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 10 monclient(hunting): picked mon.noname-a con 0x7f54201429f0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54201db2d0 con 0x7f54201429f0 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54201d4a60 con 0x7f54200a8ab0 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54201db090 con 0x7f542013f500 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.212+0000 7f5427a2b640 10 monclient(hunting): authenticate will time out at 3537.528381s 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5424fc6640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f542013f500 0x7f54201dab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5425fc8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54201429f0 0x7f54201d7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f54257c7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54200a8ab0 0x7f54201da610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5424fc6640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f542013f500 0x7f54201dab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37074/0 (socket says 172.21.15.7:37074) 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5424fc6640 1 -- 172.21.15.7:0/1782461541 learned_addr learned my addr 172.21.15.7:0/1782461541 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5424fc6640 10 monclient(hunting): get_auth_request con 0x7f542013f500 auth_method 0 2024-01-21T03:56:31.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5424fc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5424fc6640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5424fc6640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f54257c7640 10 monclient(hunting): get_auth_request con 0x7f54200a8ab0 auth_method 0 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f54257c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f54257c7640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f54257c7640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5425fc8640 10 monclient(hunting): get_auth_request con 0x7f54201429f0 auth_method 0 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5425fc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5425fc8640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5425fc8640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5424fc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.213+0000 7f5424fc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.214+0000 7f5424fc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.214+0000 7f54257c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.214+0000 7f54257c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.214+0000 7f54257c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:31.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.214+0000 7f5425fc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:31.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.214+0000 7f5425fc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:31.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.214+0000 7f5425fc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:31.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.216+0000 7f5424fc6640 10 monclient(hunting): handle_auth_done global_id 18029 payload 824 2024-01-21T03:56:31.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.216+0000 7f5424fc6640 1 -- 172.21.15.7:0/1782461541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54200a8ab0 msgr2=0x7f54201da610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.216+0000 7f5424fc6640 1 --2- 172.21.15.7:0/1782461541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54200a8ab0 0x7f54201da610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.216+0000 7f5424fc6640 1 -- 172.21.15.7:0/1782461541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54201429f0 msgr2=0x7f54201d7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.216+0000 7f5424fc6640 1 --2- 172.21.15.7:0/1782461541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54201429f0 0x7f54201d7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.216+0000 7f5424fc6640 10 monclient: _finish_hunting 0 2024-01-21T03:56:31.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.216+0000 7f5424fc6640 1 monclient: found mon.noname-c 2024-01-21T03:56:31.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.216+0000 7f5424fc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:31.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.216+0000 7f5424fc6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:56:31.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.216+0000 7f5424fc6640 1 -- 172.21.15.7:0/1782461541 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5418009a70 con 0x7f542013f500 2024-01-21T03:56:31.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.219+0000 7f5424fc6640 1 --2- 172.21.15.7:0/1782461541 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f542013f500 0x7f54201dab50 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f541000c4d0 tx=0x7f541000c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:31.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.219+0000 7f5415ffb640 10 client.?.objecter ms_handle_connect 0x7f542013f500 2024-01-21T03:56:31.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.219+0000 7f5415ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:31.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5415ffb640 1 -- 172.21.15.7:0/1782461541 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410005ce0 con 0x7f542013f500 2024-01-21T03:56:31.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5415ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5415ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5415ffb640 10 monclient: dump: 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5415ffb640 10 monclient: _finish_auth 0 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5415ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5415ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:31.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5415ffb640 1 -- 172.21.15.7:0/1782461541 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5410005e60 con 0x7f542013f500 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5415ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5415ffb640 1 -- 172.21.15.7:0/1782461541 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54100072b0 con 0x7f542013f500 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5427a2b640 5 monclient: authenticate success, global_id 18029 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5427a2b640 10 monclient: _renew_subs 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5427a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5427a2b640 1 -- 172.21.15.7:0/1782461541 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54200b0300 con 0x7f542013f500 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5415ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5416ffd640 4 set_mon_vals no callback set 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5415ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.222+0000 7f5415ffb640 10 monclient: dump: 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.225 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:31.226 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:31.226 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:31.226 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.223+0000 7f5427a2b640 10 client.18029.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:31.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.223+0000 7f5427a2b640 10 monclient: _renew_subs 2024-01-21T03:56:31.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.223+0000 7f5427a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:31.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.223+0000 7f5427a2b640 1 -- 172.21.15.7:0/1782461541 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54201d7740 con 0x7f542013f500 2024-01-21T03:56:31.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.223+0000 7f5427a2b640 1 librados: init done 2024-01-21T03:56:31.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.223+0000 7f5427a2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:31.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.223+0000 7f5427a2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:31.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.223+0000 7f5427a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:31.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.223+0000 7f5427a2b640 1 -- 172.21.15.7:0/1782461541 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5420142df0 con 0x7f542013f500 2024-01-21T03:56:31.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.241+0000 7f5415ffb640 1 -- 172.21.15.7:0/1782461541 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5410007410 con 0x7f542013f500 2024-01-21T03:56:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.242+0000 7f5415ffb640 1 --2- 172.21.15.7:0/1782461541 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f540406c8b0 0x7f540406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.242+0000 7f54257c7640 1 --2- 172.21.15.7:0/1782461541 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f540406c8b0 0x7f540406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.242+0000 7f54257c7640 10 monclient: get_auth_request con 0x7f540406c8b0 auth_method 0 2024-01-21T03:56:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.242+0000 7f5415ffb640 10 client.18029.objecter ms_handle_connect 0x7f540406c8b0 2024-01-21T03:56:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.242+0000 7f54257c7640 1 --2- 172.21.15.7:0/1782461541 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f540406c8b0 0x7f540406ed70 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f541c009b40 tx=0x7f541c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.524+0000 7f5415ffb640 1 -- 172.21.15.7:0/1782461541 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7f5410057b80 con 0x7f542013f500 2024-01-21T03:56:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.524+0000 7f5415ffb640 10 client.18029.objecter ms_dispatch 0x7f5420000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.524+0000 7f5415ffb640 3 client.18029.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.524+0000 7f5415ffb640 3 client.18029.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.524+0000 7f5415ffb640 20 client.18029.objecter dump_active .. 0 homeless 2024-01-21T03:56:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.524+0000 7f5415ffb640 1 -- 172.21.15.7:0/1782461541 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5410053ec0 con 0x7f542013f500 2024-01-21T03:56:31.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.524+0000 7f5415ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:31.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.524+0000 7f5415ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:31.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.714+0000 7f5427a2b640 1 -- 172.21.15.7:0/1782461541 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f54200a8e90 con 0x7f540406c8b0 2024-01-21T03:56:31.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.714+0000 7f5415ffb640 1 -- 172.21.15.7:0/1782461541 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f541c011a60 con 0x7f540406c8b0 2024-01-21T03:56:31.715 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 2 to deep-scrub 2024-01-21T03:56:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 10 librados: watch_flush enter 2024-01-21T03:56:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 10 librados: watch_flush exit 2024-01-21T03:56:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 20 client.18029.objecter shutdown clearing up homeless session... 2024-01-21T03:56:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 10 client.18029.objecter successfully canceled tick 2024-01-21T03:56:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 1 -- 172.21.15.7:0/1782461541 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f540406c8b0 msgr2=0x7f540406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 1 --2- 172.21.15.7:0/1782461541 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f540406c8b0 0x7f540406ed70 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f541c009b40 tx=0x7f541c009190 comp rx=0 tx=0).stop 2024-01-21T03:56:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 10 monclient: shutdown 2024-01-21T03:56:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 1 -- 172.21.15.7:0/1782461541 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f542013f500 msgr2=0x7f54201dab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 1 --2- 172.21.15.7:0/1782461541 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f542013f500 0x7f54201dab50 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f541000c4d0 tx=0x7f541000c990 comp rx=0 tx=0).stop 2024-01-21T03:56:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 1 -- 172.21.15.7:0/1782461541 shutdown_connections 2024-01-21T03:56:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 1 --2- 172.21.15.7:0/1782461541 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f540406c8b0 0x7f540406ed70 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 1 --2- 172.21.15.7:0/1782461541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54201429f0 0x7f54201d7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 1 --2- 172.21.15.7:0/1782461541 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f542013f500 0x7f54201dab50 unknown :-1 s=CLOSED pgs=3153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 1 --2- 172.21.15.7:0/1782461541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54200a8ab0 0x7f54201da610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 1 -- 172.21.15.7:0/1782461541 >> 172.21.15.7:0/1782461541 conn(0x7f5420066f50 msgr2=0x7f5420141880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 1 -- 172.21.15.7:0/1782461541 shutdown_connections 2024-01-21T03:56:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 1 -- 172.21.15.7:0/1782461541 wait complete. 2024-01-21T03:56:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.716+0000 7f5427a2b640 1 librados: shutdown 2024-01-21T03:56:31.729 INFO:tasks.ceph:Scrubbing osd.3 2024-01-21T03:56:31.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-01-21T03:56:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.833+0000 7f7dc08d7640 10 monclient: get_monmap_and_config 2024-01-21T03:56:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.833+0000 7f7dc08d7640 10 monclient: build_initial_monmap 2024-01-21T03:56:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.833+0000 7f7dc08d7640 10 monclient: monmap: 2024-01-21T03:56:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:31.834885+0000 2024-01-21T03:56:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:31.834885+0000 2024-01-21T03:56:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:31.835 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.834+0000 7f7dc08d7640 1 Processor -- start 2024-01-21T03:56:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.834+0000 7f7dc08d7640 1 -- start start 2024-01-21T03:56:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.834+0000 7f7dc08d7640 10 monclient: init 2024-01-21T03:56:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dc08d7640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dc08d7640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-01-21T03:56:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dc08d7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7dbc0a8a40 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dc08d7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7dbc0b2210 0x7f7dbc13f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dc08d7640 10 monclient(hunting): picked mon.noname-c con 0x7f7dbc0b2210 addr v2:172.21.15.177:3300/0 2024-01-21T03:56:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dc08d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7dbc0a8e20 addr v1:172.21.15.7:6789/0 2024-01-21T03:56:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dc08d7640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dbc145700 con 0x7f7dbc0a8e20 2024-01-21T03:56:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dc08d7640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dbc0aa9b0 con 0x7f7dbc0a8a40 2024-01-21T03:56:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dc08d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dc08d7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dbc0aa7b0 con 0x7f7dbc0b2210 2024-01-21T03:56:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dc08d7640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dc08d7640 10 monclient(hunting): authenticate will time out at 3538.151058s 2024-01-21T03:56:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dbad9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7dbc0a8e20 0x7f7dbc14c8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56412/0 (socket says 172.21.15.7:56412) 2024-01-21T03:56:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.835+0000 7f7dbad9d640 1 -- 172.21.15.7:0/27074928 learned_addr learned my addr 172.21.15.7:0/27074928 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.836+0000 7f7db9d9b640 1 --2- 172.21.15.7:0/27074928 >> v2:172.21.15.177:3300/0 conn(0x7f7dbc0b2210 0x7f7dbc13f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.836+0000 7f7db9d9b640 10 monclient(hunting): get_auth_request con 0x7f7dbc0b2210 auth_method 0 2024-01-21T03:56:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.836+0000 7f7db9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.836+0000 7f7db9d9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.836+0000 7f7db9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.837+0000 7f7db9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:31.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.837+0000 7f7db9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:31.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.837+0000 7f7db9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:31.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.838+0000 7f7db959a640 1 -- 172.21.15.7:0/27074928 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123280807 0 0) 0x7f7dbc145700 con 0x7f7dbc0a8e20 2024-01-21T03:56:31.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.838+0000 7f7db959a640 10 monclient(hunting): _negotiate 2024-01-21T03:56:31.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.838+0000 7f7db959a640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.838+0000 7f7db959a640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.838+0000 7f7db959a640 10 monclient(hunting): my global_id is 15873 2024-01-21T03:56:31.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.838+0000 7f7db959a640 1 -- 172.21.15.7:0/27074928 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7da8003520 con 0x7f7dbc0a8e20 2024-01-21T03:56:31.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.839+0000 7f7db9d9b640 10 monclient(hunting): handle_auth_done global_id 23182 payload 274 2024-01-21T03:56:31.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.839+0000 7f7db9d9b640 1 -- 172.21.15.7:0/27074928 >> v1:172.21.15.177:6789/0 conn(0x7f7dbc0a8a40 legacy=0x7f7dbc145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.839+0000 7f7db9d9b640 1 -- 172.21.15.7:0/27074928 >> v1:172.21.15.7:6789/0 conn(0x7f7dbc0a8e20 legacy=0x7f7dbc14c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.839+0000 7f7db9d9b640 10 monclient: _finish_hunting 0 2024-01-21T03:56:31.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.839+0000 7f7db9d9b640 1 monclient: found mon.noname-c 2024-01-21T03:56:31.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.839+0000 7f7db9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:31.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.839+0000 7f7db9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:56:31.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.839+0000 7f7db9d9b640 1 -- 172.21.15.7:0/27074928 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc0aac90 con 0x7f7dbc0b2210 2024-01-21T03:56:31.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.842+0000 7f7db9d9b640 1 --2- 172.21.15.7:0/27074928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dbc0b2210 0x7f7dbc13f1c0 secure :-1 s=READY pgs=5817 cs=0 l=1 rev1=1 crypto rx=0x7f7dac0097a0 tx=0x7f7dac02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cb220b232398ca8 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:31.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7db959a640 1 -- 172.21.15.7:0/27074928 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac030610 con 0x7f7dbc0b2210 2024-01-21T03:56:31.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7db959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:31.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7db959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:56:31.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7db959a640 10 monclient: dump: 2024-01-21T03:56:31.846 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7db959a640 10 monclient: _finish_auth 0 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7db959a640 10 monclient: _check_auth_tickets 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7db959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7db959a640 1 -- 172.21.15.7:0/27074928 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac030770 con 0x7f7dbc0b2210 2024-01-21T03:56:31.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7dc08d7640 5 monclient: authenticate success, global_id 23182 2024-01-21T03:56:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7dc08d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7db959a640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7dc08d7640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7db959a640 1 -- 172.21.15.7:0/27074928 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac030b00 con 0x7f7dbc0b2210 2024-01-21T03:56:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7db959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7dc08d7640 4 set_mon_vals no callback set 2024-01-21T03:56:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7db959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7db959a640 10 monclient: dump: 2024-01-21T03:56:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.848 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:31.849 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.849 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:31.849 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:31.849 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:31.849 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7dc08d7640 10 monclient: shutdown 2024-01-21T03:56:31.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7dc08d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:31.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7dc08d7640 1 -- 172.21.15.7:0/27074928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dbc0b2210 msgr2=0x7f7dbc13f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7dc08d7640 1 --2- 172.21.15.7:0/27074928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dbc0b2210 0x7f7dbc13f1c0 secure :-1 s=READY pgs=5817 cs=0 l=1 rev1=1 crypto rx=0x7f7dac0097a0 tx=0x7f7dac02b960 comp rx=0 tx=0).stop 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7dc08d7640 1 -- 172.21.15.7:0/27074928 shutdown_connections 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7dc08d7640 1 --2- 172.21.15.7:0/27074928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dbc0b2210 0x7f7dbc13f1c0 unknown :-1 s=CLOSED pgs=5817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7dc08d7640 1 -- 172.21.15.7:0/27074928 >> 172.21.15.7:0/27074928 conn(0x7f7dbc066f10 msgr2=0x7f7dbc0b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.846+0000 7f7dc08d7640 1 -- 172.21.15.7:0/27074928 shutdown_connections 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.847+0000 7f7dc08d7640 1 -- 172.21.15.7:0/27074928 wait complete. 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.847+0000 7f7dc08d7640 10 monclient: build_initial_monmap 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.847+0000 7f7dc08d7640 10 monclient: monmap: 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.853 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 1 librados: starting msgr at 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 1 librados: starting objecter 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 1 Processor -- start 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 1 -- start start 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 1 librados: setting wanted keys 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 1 librados: calling monclient init 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 10 monclient: init 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc0a8a40 0x7f7dbc1448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 10 monclient(hunting): picked mon.noname-a con 0x7f7dbc0a8a40 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dbc0a8e20 0x7f7dbc144110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 10 monclient(hunting): picked mon.noname-b con 0x7f7dbc0a8e20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7dbc0b2210 0x7f7dbc133b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 10 monclient(hunting): picked mon.noname-c con 0x7f7dbc0b2210 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc140bd0 con 0x7f7dbc0a8a40 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc144fa0 con 0x7f7dbc0a8e20 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc144650 con 0x7f7dbc0b2210 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.848+0000 7f7dc08d7640 10 monclient(hunting): authenticate will time out at 3538.164457s 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7dbad9d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7dbc0b2210 0x7f7dbc133b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7dba59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc0a8a40 0x7f7dbc1448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7dbad9d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7dbc0b2210 0x7f7dbc133b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37088/0 (socket says 172.21.15.7:37088) 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7dbad9d640 1 -- 172.21.15.7:0/4164490495 learned_addr learned my addr 172.21.15.7:0/4164490495 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7dbad9d640 10 monclient(hunting): get_auth_request con 0x7f7dbc0b2210 auth_method 0 2024-01-21T03:56:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7dbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7dbad9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7dbad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7dba59c640 10 monclient(hunting): get_auth_request con 0x7f7dbc0a8a40 auth_method 0 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7dba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7dba59c640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7dba59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7db9d9b640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dbc0a8e20 0x7f7dbc144110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7db9d9b640 10 monclient(hunting): get_auth_request con 0x7f7dbc0a8e20 auth_method 0 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7db9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7db9d9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.849+0000 7f7db9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.850+0000 7f7dba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.850+0000 7f7dba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.850+0000 7f7dba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.850+0000 7f7dbad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.850+0000 7f7dbad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.850+0000 7f7dbad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.850+0000 7f7db9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.850+0000 7f7db9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.850+0000 7f7db9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.852+0000 7f7dba59c640 10 monclient(hunting): handle_auth_done global_id 15876 payload 824 2024-01-21T03:56:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.853+0000 7f7dba59c640 1 -- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7dbc0b2210 msgr2=0x7f7dbc133b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.853+0000 7f7dba59c640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7dbc0b2210 0x7f7dbc133b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.853+0000 7f7dba59c640 1 -- 172.21.15.7:0/4164490495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dbc0a8e20 msgr2=0x7f7dbc144110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.853+0000 7f7dba59c640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dbc0a8e20 0x7f7dbc144110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.853+0000 7f7dba59c640 10 monclient: _finish_hunting 0 2024-01-21T03:56:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.853+0000 7f7dba59c640 1 monclient: found mon.noname-a 2024-01-21T03:56:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.853+0000 7f7dba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.853+0000 7f7dba59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:56:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.853+0000 7f7dba59c640 1 -- 172.21.15.7:0/4164490495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac009400 con 0x7f7dbc0a8a40 2024-01-21T03:56:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.853+0000 7f7dbad9d640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7dbc0b2210 0x7f7dbc133b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T03:56:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.855+0000 7f7dba59c640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc0a8a40 0x7f7dbc1448d0 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c340 tx=0x7f7da400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:31.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.855+0000 7f7db2ffd640 10 client.?.objecter ms_handle_connect 0x7f7dbc0a8a40 2024-01-21T03:56:31.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.855+0000 7f7db2ffd640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db2ffd640 1 -- 172.21.15.7:0/4164490495 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da40052d0 con 0x7f7dbc0a8a40 2024-01-21T03:56:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:56:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db2ffd640 10 monclient: dump: 2024-01-21T03:56:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db2ffd640 10 monclient: _finish_auth 0 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db2ffd640 10 monclient: _check_auth_tickets 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db2ffd640 1 -- 172.21.15.7:0/4164490495 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da4005450 con 0x7f7dbc0a8a40 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db2ffd640 1 -- 172.21.15.7:0/4164490495 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da4007260 con 0x7f7dbc0a8a40 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db2ffd640 10 monclient: dump: 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7dc08d7640 5 monclient: authenticate success, global_id 15876 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7dc08d7640 10 monclient: _renew_subs 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7dc08d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7dc08d7640 1 -- 172.21.15.7:0/4164490495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dbc134060 con 0x7f7dbc0a8a40 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.858+0000 7f7db8d99640 4 set_mon_vals no callback set 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.859+0000 7f7dc08d7640 10 client.15876.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.859+0000 7f7dc08d7640 10 monclient: _renew_subs 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.859+0000 7f7dc08d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:31.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.859+0000 7f7dc08d7640 1 -- 172.21.15.7:0/4164490495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dbc1344f0 con 0x7f7dbc0a8a40 2024-01-21T03:56:31.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.859+0000 7f7dc08d7640 1 librados: init done 2024-01-21T03:56:31.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.859+0000 7f7dc08d7640 10 client.15876.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:31.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.859+0000 7f7dc08d7640 15 client.15876.objecter _session_command_op_assign -1 1 2024-01-21T03:56:31.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.859+0000 7f7dc08d7640 20 client.15876.objecter _get_session osd=-1 returning homeless 2024-01-21T03:56:31.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.859+0000 7f7dc08d7640 10 client.15876.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:31.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.859+0000 7f7dc08d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:31.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.859+0000 7f7dc08d7640 1 -- 172.21.15.7:0/4164490495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7dbc140d40 con 0x7f7dbc0a8a40 2024-01-21T03:56:31.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.880+0000 7f7db2ffd640 1 -- 172.21.15.7:0/4164490495 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f7da40073c0 con 0x7f7dbc0a8a40 2024-01-21T03:56:31.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.881+0000 7f7db2ffd640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7d9c06c8e0 0x7f7d9c06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.881+0000 7f7db9d9b640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7d9c06c8e0 0x7f7d9c06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.881+0000 7f7db9d9b640 10 monclient: get_auth_request con 0x7f7d9c06c8e0 auth_method 0 2024-01-21T03:56:31.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.881+0000 7f7db2ffd640 10 client.15876.objecter ms_handle_connect 0x7f7d9c06c8e0 2024-01-21T03:56:31.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.881+0000 7f7db9d9b640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7d9c06c8e0 0x7f7d9c06eda0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7dac036000 tx=0x7f7dac03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 1 -- 172.21.15.7:0/4164490495 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7f7da4057cb0 con 0x7f7dbc0a8a40 2024-01-21T03:56:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 10 client.15876.objecter ms_dispatch 0x7f7dbc000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 3 client.15876.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 3 client.15876.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 10 client.15876.objecter checking command 1 2024-01-21T03:56:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] conn(0x7f7d9c0746d0 0x7f7d9c076b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 20 client.15876.objecter _get_session s=0x7f7d9c074000 osd=3 3 2024-01-21T03:56:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 20 client.15876.objecter put_session s=0x7f7d9c074000 osd=3 3 2024-01-21T03:56:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 15 client.15876.objecter _session_command_op_remove -1 1 2024-01-21T03:56:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 20 client.15876.objecter _get_session s=0x7f7d9c074000 osd=3 3 2024-01-21T03:56:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 20 client.15876.objecter get_session s=0x7f7d9c074000 osd=3 3 2024-01-21T03:56:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 15 client.15876.objecter _session_command_op_assign 3 1 2024-01-21T03:56:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 20 client.15876.objecter put_session s=0x7f7d9c074000 osd=3 4 2024-01-21T03:56:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 10 client.15876.objecter _send_command 1 2024-01-21T03:56:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 1 -- 172.21.15.7:0/4164490495 --> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d9c077240 con 0x7f7d9c0746d0 2024-01-21T03:56:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 20 client.15876.objecter dump_active .. 0 homeless 2024-01-21T03:56:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 1 -- 172.21.15.7:0/4164490495 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=1294) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7da4004f50 con 0x7f7dbc0a8a40 2024-01-21T03:56:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.883+0000 7f7db2ffd640 10 monclient: handle_get_version_reply finishing 1 version 1294 2024-01-21T03:56:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.884+0000 7f7dbad9d640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] conn(0x7f7d9c0746d0 0x7f7d9c076b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.884+0000 7f7dbad9d640 10 monclient: get_auth_request con 0x7f7d9c0746d0 auth_method 0 2024-01-21T03:56:31.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.887+0000 7f7db2ffd640 10 client.15876.objecter ms_handle_connect 0x7f7d9c0746d0 2024-01-21T03:56:31.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.887+0000 7f7dbad9d640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] conn(0x7f7d9c0746d0 0x7f7d9c076b70 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:31.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.913+0000 7f7db2ffd640 1 -- 172.21.15.7:0/4164490495 <== osd.3 v2:172.21.15.7:6802/1182044719 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7d9c077240 con 0x7f7d9c0746d0 2024-01-21T03:56:31.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.913+0000 7f7db2ffd640 10 client.15876.objecter ms_dispatch 0x7f7dbc000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:56:31.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.913+0000 7f7db2ffd640 10 client.15876.objecter _finish_command 1 = system:0 2024-01-21T03:56:31.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.913+0000 7f7db2ffd640 20 client.15876.objecter put_session s=0x7f7d9c074000 osd=3 4 2024-01-21T03:56:31.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.913+0000 7f7db2ffd640 15 client.15876.objecter _session_command_op_remove 3 1 2024-01-21T03:56:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.927+0000 7f7dc08d7640 10 client.15876.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T03:56:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.927+0000 7f7dc08d7640 15 client.15876.objecter _session_command_op_assign -1 2 2024-01-21T03:56:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.927+0000 7f7dc08d7640 20 client.15876.objecter _get_session s=0x7f7d9c074000 osd=3 3 2024-01-21T03:56:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.927+0000 7f7dc08d7640 20 client.15876.objecter put_session s=0x7f7d9c074000 osd=3 3 2024-01-21T03:56:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.927+0000 7f7dc08d7640 20 client.15876.objecter _get_session s=0x7f7d9c074000 osd=3 3 2024-01-21T03:56:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.927+0000 7f7dc08d7640 15 client.15876.objecter _session_command_op_remove -1 2 2024-01-21T03:56:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.927+0000 7f7dc08d7640 20 client.15876.objecter get_session s=0x7f7d9c074000 osd=3 3 2024-01-21T03:56:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.927+0000 7f7dc08d7640 15 client.15876.objecter _session_command_op_assign 3 2 2024-01-21T03:56:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.927+0000 7f7dc08d7640 20 client.15876.objecter put_session s=0x7f7d9c074000 osd=3 4 2024-01-21T03:56:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.927+0000 7f7dc08d7640 10 client.15876.objecter _send_command 2 2024-01-21T03:56:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.927+0000 7f7dc08d7640 1 -- 172.21.15.7:0/4164490495 --> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7dbc13f700 con 0x7f7d9c0746d0 2024-01-21T03:56:31.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.943+0000 7f7db2ffd640 1 -- 172.21.15.7:0/4164490495 <== osd.3 v2:172.21.15.7:6802/1182044719 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7db400d2d0 con 0x7f7d9c0746d0 2024-01-21T03:56:31.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.943+0000 7f7db2ffd640 10 client.15876.objecter ms_dispatch 0x7f7dbc000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:56:31.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.943+0000 7f7db2ffd640 10 client.15876.objecter _finish_command 2 = system:0 2024-01-21T03:56:31.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.943+0000 7f7db2ffd640 20 client.15876.objecter put_session s=0x7f7d9c074000 osd=3 4 2024-01-21T03:56:31.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.943+0000 7f7db2ffd640 15 client.15876.objecter _session_command_op_remove 3 2 2024-01-21T03:56:31.944 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-01-21T03:56:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T03:56:31.945 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-01-21T03:56:31.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 10 librados: watch_flush enter 2024-01-21T03:56:31.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 10 librados: watch_flush exit 2024-01-21T03:56:31.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 10 client.15876.objecter close_session for osd.3 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 1 -- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] conn(0x7f7d9c0746d0 msgr2=0x7f7d9c076b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:6802/1182044719,v1:172.21.15.7:6803/1182044719] conn(0x7f7d9c0746d0 0x7f7d9c076b70 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 20 client.15876.objecter put_session s=0x7f7d9c074000 osd=3 1 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 20 client.15876.objecter shutdown clearing up homeless session... 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 10 client.15876.objecter successfully canceled tick 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 1 -- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7d9c06c8e0 msgr2=0x7f7d9c06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f7d9c06c8e0 0x7f7d9c06eda0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7dac036000 tx=0x7f7dac03b040 comp rx=0 tx=0).stop 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 10 monclient: shutdown 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 1 -- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc0a8a40 msgr2=0x7f7dbc1448d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 1 --2- 172.21.15.7:0/4164490495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc0a8a40 0x7f7dbc1448d0 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c340 tx=0x7f7da400c800 comp rx=0 tx=0).stop 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dbad9d640 1 -- 172.21.15.7:0/4164490495 reap_dead start 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 1 -- 172.21.15.7:0/4164490495 shutdown_connections 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 1 -- 172.21.15.7:0/4164490495 >> 172.21.15.7:0/4164490495 conn(0x7f7dbc066f10 msgr2=0x7f7dbc0b6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 1 -- 172.21.15.7:0/4164490495 shutdown_connections 2024-01-21T03:56:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 1 -- 172.21.15.7:0/4164490495 wait complete. 2024-01-21T03:56:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:31.944+0000 7f7dc08d7640 1 librados: shutdown 2024-01-21T03:56:31.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-01-21T03:56:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.056+0000 7f4f3fb05640 10 monclient: get_monmap_and_config 2024-01-21T03:56:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.056+0000 7f4f3fb05640 10 monclient: build_initial_monmap 2024-01-21T03:56:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.056+0000 7f4f3fb05640 10 monclient: monmap: 2024-01-21T03:56:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:32.057425+0000 2024-01-21T03:56:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:32.057425+0000 2024-01-21T03:56:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:32.058 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:32.058 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 1 Processor -- start 2024-01-21T03:56:32.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 1 -- start start 2024-01-21T03:56:32.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 10 monclient: init 2024-01-21T03:56:32.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f3806fdb0 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f38070190 0x7f4f3813e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 10 monclient(hunting): picked mon.noname-b con 0x7f4f38070190 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f3813e930 addr v1:172.21.15.7:6789/0 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f38072380 con 0x7f4f3813e930 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3806f680 con 0x7f4f3806fdb0 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3813bf30 con 0x7f4f38070190 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.057+0000 7f4f3fb05640 10 monclient(hunting): authenticate will time out at 3538.373638s 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.058+0000 7f4f3d0a0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f38070190 0x7f4f3813e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.058+0000 7f4f3e0a2640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4f3813e930 0x7f4f38148d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56416/0 (socket says 172.21.15.7:56416) 2024-01-21T03:56:32.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.058+0000 7f4f3e0a2640 1 -- 172.21.15.7:0/4135551273 learned_addr learned my addr 172.21.15.7:0/4135551273 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:32.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.058+0000 7f4f3d0a0640 10 monclient(hunting): get_auth_request con 0x7f4f38070190 auth_method 0 2024-01-21T03:56:32.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.058+0000 7f4f3d0a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:32.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.058+0000 7f4f3d0a0640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:32.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.058+0000 7f4f3d0a0640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:32.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.059+0000 7f4f3d0a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:32.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.059+0000 7f4f3d0a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:32.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.059+0000 7f4f3d0a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:32.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.060+0000 7f4f3c89f640 1 -- 172.21.15.7:0/4135551273 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533462882 0 0) 0x7f4f3806f680 con 0x7f4f3806fdb0 2024-01-21T03:56:32.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.060+0000 7f4f3c89f640 10 monclient(hunting): _negotiate 2024-01-21T03:56:32.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.060+0000 7f4f3c89f640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:32.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.060+0000 7f4f3c89f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:32.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.060+0000 7f4f3c89f640 10 monclient(hunting): my global_id is 23188 2024-01-21T03:56:32.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.060+0000 7f4f3c89f640 1 -- 172.21.15.7:0/4135551273 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f20003520 con 0x7f4f3806fdb0 2024-01-21T03:56:32.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.060+0000 7f4f3d0a0640 10 monclient(hunting): handle_auth_done global_id 18035 payload 274 2024-01-21T03:56:32.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.060+0000 7f4f3d0a0640 1 -- 172.21.15.7:0/4135551273 >> v1:172.21.15.177:6789/0 conn(0x7f4f3806fdb0 legacy=0x7f4f38071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.060+0000 7f4f3d0a0640 1 -- 172.21.15.7:0/4135551273 >> v1:172.21.15.7:6789/0 conn(0x7f4f3813e930 legacy=0x7f4f38148d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.061+0000 7f4f3d0a0640 10 monclient: _finish_hunting 0 2024-01-21T03:56:32.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.061+0000 7f4f3d0a0640 1 monclient: found mon.noname-b 2024-01-21T03:56:32.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.061+0000 7f4f3d0a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:32.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.061+0000 7f4f3d0a0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:56:32.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.061+0000 7f4f3d0a0640 1 -- 172.21.15.7:0/4135551273 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f380adca0 con 0x7f4f38070190 2024-01-21T03:56:32.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.062+0000 7f4f3d0a0640 1 --2- 172.21.15.7:0/4135551273 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f38070190 0x7f4f3813e3f0 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0097d0 tx=0x7f4f2c02cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ba87faa180b3ff0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:32.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3c89f640 1 -- 172.21.15.7:0/4135551273 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c02da20 con 0x7f4f38070190 2024-01-21T03:56:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3c89f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3c89f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3c89f640 10 monclient: dump: 2024-01-21T03:56:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:32.068 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3c89f640 10 monclient: _finish_auth 0 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3c89f640 10 monclient: _check_auth_tickets 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3c89f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3c89f640 1 -- 172.21.15.7:0/4135551273 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2c02db80 con 0x7f4f38070190 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3fb05640 5 monclient: authenticate success, global_id 18035 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3fb05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3c89f640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3fb05640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3fb05640 4 set_mon_vals no callback set 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3c89f640 1 -- 172.21.15.7:0/4135551273 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c035440 con 0x7f4f38070190 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3c89f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3c89f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.066+0000 7f4f3c89f640 10 monclient: dump: 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.067+0000 7f4f3fb05640 10 monclient: shutdown 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.067+0000 7f4f3fb05640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.067+0000 7f4f3fb05640 1 -- 172.21.15.7:0/4135551273 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f38070190 msgr2=0x7f4f3813e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.067+0000 7f4f3fb05640 1 --2- 172.21.15.7:0/4135551273 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f38070190 0x7f4f3813e3f0 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0097d0 tx=0x7f4f2c02cc70 comp rx=0 tx=0).stop 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.067+0000 7f4f3fb05640 1 -- 172.21.15.7:0/4135551273 shutdown_connections 2024-01-21T03:56:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.067+0000 7f4f3fb05640 1 --2- 172.21.15.7:0/4135551273 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f38070190 0x7f4f3813e3f0 unknown :-1 s=CLOSED pgs=3154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.067+0000 7f4f3fb05640 1 -- 172.21.15.7:0/4135551273 >> 172.21.15.7:0/4135551273 conn(0x7f4f380b3690 msgr2=0x7f4f380b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.067+0000 7f4f3fb05640 1 -- 172.21.15.7:0/4135551273 shutdown_connections 2024-01-21T03:56:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.067+0000 7f4f3fb05640 1 -- 172.21.15.7:0/4135551273 wait complete. 2024-01-21T03:56:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.068+0000 7f4f3fb05640 10 monclient: build_initial_monmap 2024-01-21T03:56:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.068+0000 7f4f3fb05640 10 monclient: monmap: 2024-01-21T03:56:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:32.074 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.068+0000 7f4f3fb05640 1 librados: starting msgr at 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.068+0000 7f4f3fb05640 1 librados: starting objecter 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.068+0000 7f4f3fb05640 1 Processor -- start 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.068+0000 7f4f3fb05640 1 -- start start 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.068+0000 7f4f3fb05640 1 librados: setting wanted keys 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.068+0000 7f4f3fb05640 1 librados: calling monclient init 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.068+0000 7f4f3fb05640 10 monclient: init 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.068+0000 7f4f3fb05640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.068+0000 7f4f3fb05640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.068+0000 7f4f3fb05640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f3806fdb0 0x7f4f380acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.068+0000 7f4f3fb05640 10 monclient(hunting): picked mon.noname-b con 0x7f4f3806fdb0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3fb05640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f38070190 0x7f4f38134750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3fb05640 10 monclient(hunting): picked mon.noname-c con 0x7f4f38070190 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3fb05640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3813e930 0x7f4f38134c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3fb05640 10 monclient(hunting): picked mon.noname-a con 0x7f4f3813e930 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3fb05640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3fb05640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3813bd00 con 0x7f4f3813e930 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3fb05640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3fb05640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f381351d0 con 0x7f4f3806fdb0 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3fb05640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3fb05640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3813ae90 con 0x7f4f38070190 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3fb05640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3fb05640 10 monclient(hunting): authenticate will time out at 3538.384776s 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3d0a0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f38070190 0x7f4f38134750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3e0a2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3813e930 0x7f4f38134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3d8a1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f3806fdb0 0x7f4f380acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3d0a0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f38070190 0x7f4f38134750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37106/0 (socket says 172.21.15.7:37106) 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3d0a0640 1 -- 172.21.15.7:0/1193169589 learned_addr learned my addr 172.21.15.7:0/1193169589 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3d0a0640 10 monclient(hunting): get_auth_request con 0x7f4f38070190 auth_method 0 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3d0a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3d0a0640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3d0a0640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3d8a1640 10 monclient(hunting): get_auth_request con 0x7f4f3806fdb0 auth_method 0 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3d8a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3d8a1640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3d8a1640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3e0a2640 10 monclient(hunting): get_auth_request con 0x7f4f3813e930 auth_method 0 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3e0a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3e0a2640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.069+0000 7f4f3e0a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.070+0000 7f4f3d0a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.070+0000 7f4f3d0a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.070+0000 7f4f3d0a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:32.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.070+0000 7f4f3d8a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.070+0000 7f4f3d8a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.070+0000 7f4f3d8a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.070+0000 7f4f3e0a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.070+0000 7f4f3e0a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.070+0000 7f4f3e0a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.073+0000 7f4f3d8a1640 10 monclient(hunting): handle_auth_done global_id 23191 payload 824 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.073+0000 7f4f3d8a1640 1 -- 172.21.15.7:0/1193169589 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f38070190 msgr2=0x7f4f38134750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.073+0000 7f4f3d8a1640 1 --2- 172.21.15.7:0/1193169589 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f38070190 0x7f4f38134750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.073+0000 7f4f3d8a1640 1 -- 172.21.15.7:0/1193169589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3813e930 msgr2=0x7f4f38134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.073+0000 7f4f3d8a1640 1 --2- 172.21.15.7:0/1193169589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3813e930 0x7f4f38134c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.073+0000 7f4f3d8a1640 10 monclient: _finish_hunting 0 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.073+0000 7f4f3d8a1640 1 monclient: found mon.noname-b 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.073+0000 7f4f3d8a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.073+0000 7f4f3d8a1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.073+0000 7f4f3d8a1640 1 -- 172.21.15.7:0/1193169589 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c009400 con 0x7f4f3806fdb0 2024-01-21T03:56:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.073+0000 7f4f3d0a0640 1 --2- 172.21.15.7:0/1193169589 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f38070190 0x7f4f38134750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T03:56:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.075+0000 7f4f3d8a1640 1 --2- 172.21.15.7:0/1193169589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f3806fdb0 0x7f4f380acb60 secure :-1 s=READY pgs=5819 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000c400 tx=0x7f4f3000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.075+0000 7f4f367fc640 10 client.?.objecter ms_handle_connect 0x7f4f3806fdb0 2024-01-21T03:56:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.075+0000 7f4f367fc640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f367fc640 1 -- 172.21.15.7:0/1193169589 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f300052d0 con 0x7f4f3806fdb0 2024-01-21T03:56:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f367fc640 10 monclient: dump: 2024-01-21T03:56:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:32.081 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f367fc640 10 monclient: _finish_auth 0 2024-01-21T03:56:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f367fc640 10 monclient: _check_auth_tickets 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f367fc640 1 -- 172.21.15.7:0/1193169589 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f30005430 con 0x7f4f3806fdb0 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f367fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f3fb05640 5 monclient: authenticate success, global_id 23191 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f367fc640 1 -- 172.21.15.7:0/1193169589 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f30003830 con 0x7f4f3806fdb0 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f367fc640 10 monclient: dump: 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:32.083 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f37fff640 4 set_mon_vals no callback set 2024-01-21T03:56:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f3fb05640 10 monclient: _renew_subs 2024-01-21T03:56:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f3fb05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f3fb05640 1 -- 172.21.15.7:0/1193169589 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3813b120 con 0x7f4f3806fdb0 2024-01-21T03:56:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f3fb05640 10 client.23191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f3fb05640 10 monclient: _renew_subs 2024-01-21T03:56:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f3fb05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f3fb05640 1 -- 172.21.15.7:0/1193169589 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3813b5b0 con 0x7f4f3806fdb0 2024-01-21T03:56:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.079+0000 7f4f3fb05640 1 librados: init done 2024-01-21T03:56:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.080+0000 7f4f3fb05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.080+0000 7f4f3fb05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.080+0000 7f4f3fb05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.080+0000 7f4f3fb05640 1 -- 172.21.15.7:0/1193169589 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f38070d10 con 0x7f4f3806fdb0 2024-01-21T03:56:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.322+0000 7f4f367fc640 1 -- 172.21.15.7:0/1193169589 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4f30003990 con 0x7f4f3806fdb0 2024-01-21T03:56:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.322+0000 7f4f367fc640 1 --2- 172.21.15.7:0/1193169589 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4f2406c8b0 0x7f4f2406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.322+0000 7f4f3d0a0640 1 --2- 172.21.15.7:0/1193169589 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4f2406c8b0 0x7f4f2406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.322+0000 7f4f3d0a0640 10 monclient: get_auth_request con 0x7f4f2406c8b0 auth_method 0 2024-01-21T03:56:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.323+0000 7f4f367fc640 10 client.23191.objecter ms_handle_connect 0x7f4f2406c8b0 2024-01-21T03:56:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.323+0000 7f4f3d0a0640 1 --2- 172.21.15.7:0/1193169589 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4f2406c8b0 0x7f4f2406ed70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c000c00 tx=0x7f4f2c031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.325+0000 7f4f367fc640 1 -- 172.21.15.7:0/1193169589 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7f4f30057c90 con 0x7f4f3806fdb0 2024-01-21T03:56:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.325+0000 7f4f367fc640 10 client.23191.objecter ms_dispatch 0x7f4f38000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.325+0000 7f4f367fc640 3 client.23191.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.325+0000 7f4f367fc640 3 client.23191.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.326+0000 7f4f367fc640 20 client.23191.objecter dump_active .. 0 homeless 2024-01-21T03:56:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.335+0000 7f4f367fc640 1 -- 172.21.15.7:0/1193169589 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4f30053fd0 con 0x7f4f3806fdb0 2024-01-21T03:56:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.335+0000 7f4f367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.335+0000 7f4f367fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:32.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.515+0000 7f4f3fb05640 1 -- 172.21.15.7:0/1193169589 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f3804e970 con 0x7f4f2406c8b0 2024-01-21T03:56:32.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.516+0000 7f4f367fc640 1 -- 172.21.15.7:0/1193169589 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4f3804e970 con 0x7f4f2406c8b0 2024-01-21T03:56:32.517 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 3 to deep-scrub 2024-01-21T03:56:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 10 librados: watch_flush enter 2024-01-21T03:56:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 10 librados: watch_flush exit 2024-01-21T03:56:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 20 client.23191.objecter shutdown clearing up homeless session... 2024-01-21T03:56:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 10 client.23191.objecter successfully canceled tick 2024-01-21T03:56:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 1 -- 172.21.15.7:0/1193169589 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4f2406c8b0 msgr2=0x7f4f2406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 1 --2- 172.21.15.7:0/1193169589 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4f2406c8b0 0x7f4f2406ed70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c000c00 tx=0x7f4f2c031000 comp rx=0 tx=0).stop 2024-01-21T03:56:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 10 monclient: shutdown 2024-01-21T03:56:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 1 -- 172.21.15.7:0/1193169589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f3806fdb0 msgr2=0x7f4f380acb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 1 --2- 172.21.15.7:0/1193169589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f3806fdb0 0x7f4f380acb60 secure :-1 s=READY pgs=5819 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000c400 tx=0x7f4f3000c8c0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 1 -- 172.21.15.7:0/1193169589 shutdown_connections 2024-01-21T03:56:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 1 --2- 172.21.15.7:0/1193169589 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4f2406c8b0 0x7f4f2406ed70 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 1 --2- 172.21.15.7:0/1193169589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3813e930 0x7f4f38134c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 1 --2- 172.21.15.7:0/1193169589 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4f38070190 0x7f4f38134750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 1 --2- 172.21.15.7:0/1193169589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f3806fdb0 0x7f4f380acb60 unknown :-1 s=CLOSED pgs=5819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 1 -- 172.21.15.7:0/1193169589 >> 172.21.15.7:0/1193169589 conn(0x7f4f380b3690 msgr2=0x7f4f380b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 1 -- 172.21.15.7:0/1193169589 shutdown_connections 2024-01-21T03:56:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 1 -- 172.21.15.7:0/1193169589 wait complete. 2024-01-21T03:56:32.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.518+0000 7f4f3fb05640 1 librados: shutdown 2024-01-21T03:56:32.529 INFO:tasks.ceph:Scrubbing osd.4 2024-01-21T03:56:32.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-01-21T03:56:32.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.638+0000 7f4c225a9640 10 monclient: get_monmap_and_config 2024-01-21T03:56:32.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.638+0000 7f4c225a9640 10 monclient: build_initial_monmap 2024-01-21T03:56:32.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.638+0000 7f4c225a9640 10 monclient: monmap: 2024-01-21T03:56:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:32.639540+0000 2024-01-21T03:56:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:32.639540+0000 2024-01-21T03:56:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:32.642 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.639+0000 7f4c225a9640 1 Processor -- start 2024-01-21T03:56:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.639+0000 7f4c225a9640 1 -- start start 2024-01-21T03:56:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.639+0000 7f4c225a9640 10 monclient: init 2024-01-21T03:56:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-01-21T03:56:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c1c0a8b20 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c1c0a9d10 0x7f4c1c13f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 10 monclient(hunting): picked mon.noname-b con 0x7f4c1c0a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4c1c0b2310 0x7f4c1c13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 10 monclient(hunting): picked mon.noname-a con 0x7f4c1c0b2310 addr v2:172.21.15.7:3300/0 2024-01-21T03:56:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1c1456f0 con 0x7f4c1c0a8b20 2024-01-21T03:56:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c1c0a98c0 con 0x7f4c1c0b2310 2024-01-21T03:56:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c13fb00 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c225a9640 10 monclient(hunting): authenticate will time out at 3538.955853s 2024-01-21T03:56:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c13fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c1c0a9d10 0x7f4c1c13f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c1bfff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f4c1c0a8b20 0x7f4c1c145b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:42604/0 (socket says 172.21.15.7:42604) 2024-01-21T03:56:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c1bfff640 1 -- 172.21.15.7:0/1841451234 learned_addr learned my addr 172.21.15.7:0/1841451234 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c13fff640 10 monclient(hunting): get_auth_request con 0x7f4c1c0a9d10 auth_method 0 2024-01-21T03:56:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c13fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.640+0000 7f4c13fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.641+0000 7f4c13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.641+0000 7f4c13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.641+0000 7f4c13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.643+0000 7f4c13fff640 10 monclient(hunting): handle_auth_done global_id 18041 payload 274 2024-01-21T03:56:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.643+0000 7f4c13fff640 1 -- 172.21.15.7:0/1841451234 >> v2:172.21.15.7:3300/0 conn(0x7f4c1c0b2310 msgr2=0x7f4c1c13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.643+0000 7f4c13fff640 1 --2- 172.21.15.7:0/1841451234 >> v2:172.21.15.7:3300/0 conn(0x7f4c1c0b2310 0x7f4c1c13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.643+0000 7f4c13fff640 1 -- 172.21.15.7:0/1841451234 >> v1:172.21.15.177:6789/0 conn(0x7f4c1c0a8b20 legacy=0x7f4c1c145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.643+0000 7f4c13fff640 10 monclient: _finish_hunting 0 2024-01-21T03:56:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.643+0000 7f4c13fff640 1 monclient: found mon.noname-b 2024-01-21T03:56:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.643+0000 7f4c13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:32.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.643+0000 7f4c13fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:56:32.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.643+0000 7f4c13fff640 1 -- 172.21.15.7:0/1841451234 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1c146290 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.645+0000 7f4c13fff640 1 --2- 172.21.15.7:0/1841451234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c1c0a9d10 0x7f4c1c13f080 secure :-1 s=READY pgs=3155 cs=0 l=1 rev1=1 crypto rx=0x7f4c040097a0 tx=0x7f4c0402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4e3b589afba8c92 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c1b7fe640 1 -- 172.21.15.7:0/1841451234 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04031830 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c1b7fe640 10 monclient: dump: 2024-01-21T03:56:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:32.650 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c1b7fe640 10 monclient: _finish_auth 0 2024-01-21T03:56:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c1b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:56:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c1b7fe640 1 -- 172.21.15.7:0/1841451234 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c04031990 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c1b7fe640 1 -- 172.21.15.7:0/1841451234 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04031d20 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c225a9640 5 monclient: authenticate success, global_id 18041 2024-01-21T03:56:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c225a9640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c225a9640 4 set_mon_vals no callback set 2024-01-21T03:56:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.648+0000 7f4c1b7fe640 10 monclient: dump: 2024-01-21T03:56:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:32.652 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.649+0000 7f4c225a9640 10 monclient: shutdown 2024-01-21T03:56:32.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.649+0000 7f4c225a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:32.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.649+0000 7f4c225a9640 1 -- 172.21.15.7:0/1841451234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c1c0a9d10 msgr2=0x7f4c1c13f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.649+0000 7f4c225a9640 1 --2- 172.21.15.7:0/1841451234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c1c0a9d10 0x7f4c1c13f080 secure :-1 s=READY pgs=3155 cs=0 l=1 rev1=1 crypto rx=0x7f4c040097a0 tx=0x7f4c0402b8d0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.649+0000 7f4c225a9640 1 -- 172.21.15.7:0/1841451234 shutdown_connections 2024-01-21T03:56:32.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.649+0000 7f4c225a9640 1 --2- 172.21.15.7:0/1841451234 >> v2:172.21.15.7:3300/0 conn(0x7f4c1c0b2310 0x7f4c1c13f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.649+0000 7f4c225a9640 1 --2- 172.21.15.7:0/1841451234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c1c0a9d10 0x7f4c1c13f080 unknown :-1 s=CLOSED pgs=3155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.649+0000 7f4c225a9640 1 -- 172.21.15.7:0/1841451234 >> 172.21.15.7:0/1841451234 conn(0x7f4c1c066fb0 msgr2=0x7f4c1c0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.649+0000 7f4c225a9640 1 -- 172.21.15.7:0/1841451234 shutdown_connections 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.649+0000 7f4c225a9640 1 -- 172.21.15.7:0/1841451234 wait complete. 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.650+0000 7f4c225a9640 10 monclient: build_initial_monmap 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.650+0000 7f4c225a9640 10 monclient: monmap: 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:32.657 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.651+0000 7f4c225a9640 1 librados: starting msgr at 2024-01-21T03:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.651+0000 7f4c225a9640 1 librados: starting objecter 2024-01-21T03:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.651+0000 7f4c225a9640 1 Processor -- start 2024-01-21T03:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.651+0000 7f4c225a9640 1 -- start start 2024-01-21T03:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.651+0000 7f4c225a9640 1 librados: setting wanted keys 2024-01-21T03:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.651+0000 7f4c225a9640 1 librados: calling monclient init 2024-01-21T03:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.651+0000 7f4c225a9640 10 monclient: init 2024-01-21T03:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T03:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c1c0a8b20 0x7f4c1c144260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 10 monclient(hunting): picked mon.noname-b con 0x7f4c1c0a8b20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c1c0a9d10 0x7f4c1c12fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 10 monclient(hunting): picked mon.noname-c con 0x7f4c1c0a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c1c0b2310 0x7f4c1c1309e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 10 monclient(hunting): picked mon.noname-a con 0x7f4c1c0b2310 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c1305c0 con 0x7f4c1c0b2310 2024-01-21T03:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c12f510 con 0x7f4c1c0a8b20 2024-01-21T03:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c12f7f0 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c225a9640 10 monclient(hunting): authenticate will time out at 3538.968240s 2024-01-21T03:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.652+0000 7f4c13fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c1c0a9d10 0x7f4c1c12fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c1bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c1c0a8b20 0x7f4c1c144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c13fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c1c0a9d10 0x7f4c1c12fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37124/0 (socket says 172.21.15.7:37124) 2024-01-21T03:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c13fff640 1 -- 172.21.15.7:0/943560498 learned_addr learned my addr 172.21.15.7:0/943560498 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c13fff640 10 monclient(hunting): get_auth_request con 0x7f4c1c0a9d10 auth_method 0 2024-01-21T03:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c13fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c13fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c1bfff640 10 monclient(hunting): get_auth_request con 0x7f4c1c0a8b20 auth_method 0 2024-01-21T03:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c1bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:32.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:32.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:32.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:32.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:32.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.653+0000 7f4c1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:32.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.654+0000 7f4c1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:32.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.656+0000 7f4c13fff640 10 monclient(hunting): handle_auth_done global_id 18044 payload 824 2024-01-21T03:56:32.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.656+0000 7f4c13fff640 1 -- 172.21.15.7:0/943560498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c1c0a8b20 msgr2=0x7f4c1c144260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.656+0000 7f4c13fff640 1 --2- 172.21.15.7:0/943560498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c1c0a8b20 0x7f4c1c144260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.656+0000 7f4c13fff640 1 -- 172.21.15.7:0/943560498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c1c0b2310 msgr2=0x7f4c1c1309e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.656+0000 7f4c13fff640 1 --2- 172.21.15.7:0/943560498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c1c0b2310 0x7f4c1c1309e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.656+0000 7f4c13fff640 10 monclient: _finish_hunting 0 2024-01-21T03:56:32.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.656+0000 7f4c13fff640 1 monclient: found mon.noname-c 2024-01-21T03:56:32.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.656+0000 7f4c13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:32.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.656+0000 7f4c13fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:56:32.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.656+0000 7f4c13fff640 1 -- 172.21.15.7:0/943560498 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c04009400 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.658+0000 7f4c13fff640 1 --2- 172.21.15.7:0/943560498 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c1c0a9d10 0x7f4c1c12fc10 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f4c040098c0 tx=0x7f4c040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:32.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.658+0000 7f4c197fa640 10 client.?.objecter ms_handle_connect 0x7f4c1c0a9d10 2024-01-21T03:56:32.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.658+0000 7f4c197fa640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:32.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c197fa640 1 -- 172.21.15.7:0/943560498 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0403bb70 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:32.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c197fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:56:32.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c197fa640 10 monclient: dump: 2024-01-21T03:56:32.665 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:32.665 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.665 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.665 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.665 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:32.665 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.665 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:32.665 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:32.666 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:32.666 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c197fa640 10 monclient: _finish_auth 0 2024-01-21T03:56:32.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c197fa640 10 monclient: _check_auth_tickets 2024-01-21T03:56:32.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:32.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c197fa640 1 -- 172.21.15.7:0/943560498 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0403bcf0 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c225a9640 5 monclient: authenticate success, global_id 18044 2024-01-21T03:56:32.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c225a9640 10 monclient: _renew_subs 2024-01-21T03:56:32.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c225a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:32.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c225a9640 1 -- 172.21.15.7:0/943560498 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1c130150 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c197fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:32.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c197fa640 1 -- 172.21.15.7:0/943560498 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04045630 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:32.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:32.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c197fa640 10 monclient: dump: 2024-01-21T03:56:32.667 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:32.667 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.667 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.667 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.667 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:32.667 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.667 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:32.667 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:32.667 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:32.667 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c1affd640 4 set_mon_vals no callback set 2024-01-21T03:56:32.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c225a9640 10 client.18044.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:32.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c225a9640 10 monclient: _renew_subs 2024-01-21T03:56:32.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c225a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:32.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c225a9640 1 -- 172.21.15.7:0/943560498 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1c13ccf0 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.662+0000 7f4c225a9640 1 librados: init done 2024-01-21T03:56:32.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.663+0000 7f4c225a9640 10 client.18044.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:32.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.663+0000 7f4c225a9640 15 client.18044.objecter _session_command_op_assign -1 1 2024-01-21T03:56:32.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.663+0000 7f4c225a9640 20 client.18044.objecter _get_session osd=-1 returning homeless 2024-01-21T03:56:32.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.663+0000 7f4c225a9640 10 client.18044.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:32.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.663+0000 7f4c225a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:32.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.663+0000 7f4c225a9640 1 -- 172.21.15.7:0/943560498 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c1c0b2940 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.683+0000 7f4c197fa640 1 -- 172.21.15.7:0/943560498 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4c0404e450 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.683+0000 7f4c197fa640 1 --2- 172.21.15.7:0/943560498 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4bfc06c8b0 0x7f4bfc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.683+0000 7f4c1bfff640 1 --2- 172.21.15.7:0/943560498 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4bfc06c8b0 0x7f4bfc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.683+0000 7f4c1bfff640 10 monclient: get_auth_request con 0x7f4bfc06c8b0 auth_method 0 2024-01-21T03:56:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.683+0000 7f4c1bfff640 1 --2- 172.21.15.7:0/943560498 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4bfc06c8b0 0x7f4bfc06ed70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c007fd0 tx=0x7f4c0c007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.683+0000 7f4c197fa640 10 client.18044.objecter ms_handle_connect 0x7f4bfc06c8b0 2024-01-21T03:56:32.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 1 -- 172.21.15.7:0/943560498 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7f4c0407b3b0 con 0x7f4c1c0a9d10 2024-01-21T03:56:32.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 10 client.18044.objecter ms_dispatch 0x7f4c1c000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:32.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 3 client.18044.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:32.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 3 client.18044.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:32.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 10 client.18044.objecter checking command 1 2024-01-21T03:56:32.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 1 --2- 172.21.15.7:0/943560498 >> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] conn(0x7f4bfc0746a0 0x7f4bfc076b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 20 client.18044.objecter _get_session s=0x7f4bfc073fd0 osd=4 3 2024-01-21T03:56:32.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 20 client.18044.objecter put_session s=0x7f4bfc073fd0 osd=4 3 2024-01-21T03:56:32.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 15 client.18044.objecter _session_command_op_remove -1 1 2024-01-21T03:56:32.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 20 client.18044.objecter _get_session s=0x7f4bfc073fd0 osd=4 3 2024-01-21T03:56:32.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 20 client.18044.objecter get_session s=0x7f4bfc073fd0 osd=4 3 2024-01-21T03:56:32.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 15 client.18044.objecter _session_command_op_assign 4 1 2024-01-21T03:56:32.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 20 client.18044.objecter put_session s=0x7f4bfc073fd0 osd=4 4 2024-01-21T03:56:32.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 10 client.18044.objecter _send_command 1 2024-01-21T03:56:32.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 1 -- 172.21.15.7:0/943560498 --> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bfc077210 con 0x7f4bfc0746a0 2024-01-21T03:56:32.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.686+0000 7f4c197fa640 20 client.18044.objecter dump_active .. 0 homeless 2024-01-21T03:56:32.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.687+0000 7f4c20b46640 1 --2- 172.21.15.7:0/943560498 >> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] conn(0x7f4bfc0746a0 0x7f4bfc076b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.687+0000 7f4c20b46640 10 monclient: get_auth_request con 0x7f4bfc0746a0 auth_method 0 2024-01-21T03:56:32.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.690+0000 7f4c20b46640 1 --2- 172.21.15.7:0/943560498 >> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] conn(0x7f4bfc0746a0 0x7f4bfc076b40 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:32.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.690+0000 7f4c197fa640 10 client.18044.objecter ms_handle_connect 0x7f4bfc0746a0 2024-01-21T03:56:32.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.715+0000 7f4c197fa640 1 -- 172.21.15.7:0/943560498 <== osd.4 v2:172.21.15.177:6808/3975537014 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4bfc077210 con 0x7f4bfc0746a0 2024-01-21T03:56:32.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.715+0000 7f4c197fa640 10 client.18044.objecter ms_dispatch 0x7f4c1c000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:56:32.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.715+0000 7f4c197fa640 10 client.18044.objecter _finish_command 1 = system:0 2024-01-21T03:56:32.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.715+0000 7f4c197fa640 20 client.18044.objecter put_session s=0x7f4bfc073fd0 osd=4 4 2024-01-21T03:56:32.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.715+0000 7f4c197fa640 15 client.18044.objecter _session_command_op_remove 4 1 2024-01-21T03:56:32.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.728+0000 7f4c225a9640 10 client.18044.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T03:56:32.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.728+0000 7f4c225a9640 15 client.18044.objecter _session_command_op_assign -1 2 2024-01-21T03:56:32.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.728+0000 7f4c225a9640 20 client.18044.objecter _get_session s=0x7f4bfc073fd0 osd=4 3 2024-01-21T03:56:32.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.728+0000 7f4c225a9640 20 client.18044.objecter put_session s=0x7f4bfc073fd0 osd=4 3 2024-01-21T03:56:32.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.728+0000 7f4c225a9640 20 client.18044.objecter _get_session s=0x7f4bfc073fd0 osd=4 3 2024-01-21T03:56:32.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.728+0000 7f4c225a9640 15 client.18044.objecter _session_command_op_remove -1 2 2024-01-21T03:56:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.728+0000 7f4c225a9640 20 client.18044.objecter get_session s=0x7f4bfc073fd0 osd=4 3 2024-01-21T03:56:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.728+0000 7f4c225a9640 15 client.18044.objecter _session_command_op_assign 4 2 2024-01-21T03:56:32.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.728+0000 7f4c225a9640 20 client.18044.objecter put_session s=0x7f4bfc073fd0 osd=4 4 2024-01-21T03:56:32.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.728+0000 7f4c225a9640 10 client.18044.objecter _send_command 2 2024-01-21T03:56:32.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.728+0000 7f4c225a9640 1 -- 172.21.15.7:0/943560498 --> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4c1c0a9110 con 0x7f4bfc0746a0 2024-01-21T03:56:32.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.742+0000 7f4c197fa640 1 -- 172.21.15.7:0/943560498 <== osd.4 v2:172.21.15.177:6808/3975537014 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4c1400ac40 con 0x7f4bfc0746a0 2024-01-21T03:56:32.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.742+0000 7f4c197fa640 10 client.18044.objecter ms_dispatch 0x7f4c1c000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:56:32.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.742+0000 7f4c197fa640 10 client.18044.objecter _finish_command 2 = system:0 2024-01-21T03:56:32.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.742+0000 7f4c197fa640 20 client.18044.objecter put_session s=0x7f4bfc073fd0 osd=4 4 2024-01-21T03:56:32.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.742+0000 7f4c197fa640 15 client.18044.objecter _session_command_op_remove 4 2 2024-01-21T03:56:32.743 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-01-21T03:56:32.743 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T03:56:32.743 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 10 librados: watch_flush enter 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 10 librados: watch_flush exit 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 10 client.18044.objecter close_session for osd.4 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 1 -- 172.21.15.7:0/943560498 >> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] conn(0x7f4bfc0746a0 msgr2=0x7f4bfc076b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 1 --2- 172.21.15.7:0/943560498 >> [v2:172.21.15.177:6808/3975537014,v1:172.21.15.177:6809/3975537014] conn(0x7f4bfc0746a0 0x7f4bfc076b40 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 20 client.18044.objecter put_session s=0x7f4bfc073fd0 osd=4 1 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 20 client.18044.objecter shutdown clearing up homeless session... 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 10 client.18044.objecter successfully canceled tick 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 1 -- 172.21.15.7:0/943560498 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4bfc06c8b0 msgr2=0x7f4bfc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 1 --2- 172.21.15.7:0/943560498 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f4bfc06c8b0 0x7f4bfc06ed70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c007fd0 tx=0x7f4c0c007e80 comp rx=0 tx=0).stop 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 10 monclient: shutdown 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 20 monclient: shutdown canceling and discarding version request 1 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 1 -- 172.21.15.7:0/943560498 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c1c0a9d10 msgr2=0x7f4c1c12fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 1 --2- 172.21.15.7:0/943560498 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c1c0a9d10 0x7f4c1c12fc10 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f4c040098c0 tx=0x7f4c040046c0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c20b46640 1 -- 172.21.15.7:0/943560498 reap_dead start 2024-01-21T03:56:32.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 1 -- 172.21.15.7:0/943560498 shutdown_connections 2024-01-21T03:56:32.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 1 -- 172.21.15.7:0/943560498 >> 172.21.15.7:0/943560498 conn(0x7f4c1c066fb0 msgr2=0x7f4c1c13ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:32.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 1 -- 172.21.15.7:0/943560498 shutdown_connections 2024-01-21T03:56:32.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 1 -- 172.21.15.7:0/943560498 wait complete. 2024-01-21T03:56:32.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.743+0000 7f4c225a9640 1 librados: shutdown 2024-01-21T03:56:32.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-01-21T03:56:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.856+0000 7fd35db6c640 10 monclient: get_monmap_and_config 2024-01-21T03:56:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.856+0000 7fd35db6c640 10 monclient: build_initial_monmap 2024-01-21T03:56:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.856+0000 7fd35db6c640 10 monclient: monmap: 2024-01-21T03:56:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:32.857346+0000 2024-01-21T03:56:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:32.857346+0000 2024-01-21T03:56:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:32.858 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:32.858 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:32.858 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 1 Processor -- start 2024-01-21T03:56:32.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 1 -- start start 2024-01-21T03:56:32.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 10 monclient: init 2024-01-21T03:56:32.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:32.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-01-21T03:56:32.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3580a8ab0 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:32.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3580a8e90 0x7fd35813f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 10 monclient(hunting): picked mon.noname-b con 0x7fd3580a8e90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:32.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd3580b2310 0x7fd35813f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 10 monclient(hunting): picked mon.noname-a con 0x7fd3580b2310 addr v2:172.21.15.7:3300/0 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd358142dc0 con 0x7fd3580a8ab0 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3580aa820 con 0x7fd3580b2310 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3580aaa20 con 0x7fd3580a8e90 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.857+0000 7fd35db6c640 10 monclient(hunting): authenticate will time out at 3539.173504s 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.858+0000 7fd356ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3580a8e90 0x7fd35813f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.858+0000 7fd357fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd3580b2310 0x7fd35813f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.858+0000 7fd3577fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fd3580a8ab0 0x7fd3581426b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:42614/0 (socket says 172.21.15.7:42614) 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.858+0000 7fd3577fe640 1 -- 172.21.15.7:0/1338368140 learned_addr learned my addr 172.21.15.7:0/1338368140 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.858+0000 7fd356ffd640 10 monclient(hunting): get_auth_request con 0x7fd3580a8e90 auth_method 0 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.858+0000 7fd356ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.858+0000 7fd356ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.858+0000 7fd356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.858+0000 7fd357fff640 10 monclient(hunting): get_auth_request con 0x7fd3580b2310 auth_method 0 2024-01-21T03:56:32.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.858+0000 7fd357fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:32.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.858+0000 7fd357fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:32.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.858+0000 7fd357fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:32.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.859+0000 7fd356ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:32.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.859+0000 7fd356ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:32.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.859+0000 7fd356ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:32.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.859+0000 7fd357fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:32.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.859+0000 7fd357fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:32.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.859+0000 7fd357fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:32.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.860+0000 7fd356ffd640 10 monclient(hunting): handle_auth_done global_id 18047 payload 274 2024-01-21T03:56:32.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.860+0000 7fd356ffd640 1 -- 172.21.15.7:0/1338368140 >> v2:172.21.15.7:3300/0 conn(0x7fd3580b2310 msgr2=0x7fd35813f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:32.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.860+0000 7fd356ffd640 1 --2- 172.21.15.7:0/1338368140 >> v2:172.21.15.7:3300/0 conn(0x7fd3580b2310 0x7fd35813f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.860+0000 7fd356ffd640 1 -- 172.21.15.7:0/1338368140 >> v1:172.21.15.177:6789/0 conn(0x7fd3580a8ab0 legacy=0x7fd3581426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.860+0000 7fd356ffd640 10 monclient: _finish_hunting 0 2024-01-21T03:56:32.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.860+0000 7fd356ffd640 1 monclient: found mon.noname-b 2024-01-21T03:56:32.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.860+0000 7fd356ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:32.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.860+0000 7fd356ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:56:32.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.860+0000 7fd356ffd640 1 -- 172.21.15.7:0/1338368140 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3580aad00 con 0x7fd3580a8e90 2024-01-21T03:56:32.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.862+0000 7fd356ffd640 1 --2- 172.21.15.7:0/1338368140 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3580a8e90 0x7fd35813f240 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7fd34c0097a0 tx=0x7fd34c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0d40bcc9f57d12f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd3567fc640 1 -- 172.21.15.7:0/1338368140 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c02e070 con 0x7fd3580a8e90 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd3567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd3567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd3567fc640 10 monclient: dump: 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd3567fc640 10 monclient: _finish_auth 0 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd3567fc640 10 monclient: _check_auth_tickets 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd3567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd3567fc640 1 -- 172.21.15.7:0/1338368140 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c0315a0 con 0x7fd3580a8e90 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd35db6c640 5 monclient: authenticate success, global_id 18047 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd35db6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd3567fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd35db6c640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd3567fc640 1 -- 172.21.15.7:0/1338368140 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c031930 con 0x7fd3580a8e90 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd35db6c640 4 set_mon_vals no callback set 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd3567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd3567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.866+0000 7fd3567fc640 10 monclient: dump: 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:32.868 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.869 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.869 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.869 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:32.869 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.869 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:32.869 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:32.869 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:32.869 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.867+0000 7fd35db6c640 10 monclient: shutdown 2024-01-21T03:56:32.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.867+0000 7fd35db6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.867+0000 7fd35db6c640 1 -- 172.21.15.7:0/1338368140 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3580a8e90 msgr2=0x7fd35813f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.867+0000 7fd35db6c640 1 --2- 172.21.15.7:0/1338368140 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3580a8e90 0x7fd35813f240 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7fd34c0097a0 tx=0x7fd34c02b840 comp rx=0 tx=0).stop 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.867+0000 7fd35db6c640 1 -- 172.21.15.7:0/1338368140 shutdown_connections 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.867+0000 7fd35db6c640 1 --2- 172.21.15.7:0/1338368140 >> v2:172.21.15.7:3300/0 conn(0x7fd3580b2310 0x7fd35813f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.867+0000 7fd35db6c640 1 --2- 172.21.15.7:0/1338368140 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3580a8e90 0x7fd35813f240 unknown :-1 s=CLOSED pgs=3157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.867+0000 7fd35db6c640 1 -- 172.21.15.7:0/1338368140 >> 172.21.15.7:0/1338368140 conn(0x7fd358066f00 msgr2=0x7fd3580b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.867+0000 7fd35db6c640 1 -- 172.21.15.7:0/1338368140 shutdown_connections 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.867+0000 7fd35db6c640 1 -- 172.21.15.7:0/1338368140 wait complete. 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.867+0000 7fd35db6c640 10 monclient: build_initial_monmap 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.867+0000 7fd35db6c640 10 monclient: monmap: 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.868+0000 7fd35db6c640 1 librados: starting msgr at 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.868+0000 7fd35db6c640 1 librados: starting objecter 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.868+0000 7fd35db6c640 1 Processor -- start 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.868+0000 7fd35db6c640 1 -- start start 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.868+0000 7fd35db6c640 1 librados: setting wanted keys 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.868+0000 7fd35db6c640 1 librados: calling monclient init 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.868+0000 7fd35db6c640 10 monclient: init 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.868+0000 7fd35db6c640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.868+0000 7fd35db6c640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.868+0000 7fd35db6c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3580a8ab0 0x7fd3580ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.868+0000 7fd35db6c640 10 monclient(hunting): picked mon.noname-a con 0x7fd3580a8ab0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd35db6c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3580a8e90 0x7fd3581389f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd35db6c640 10 monclient(hunting): picked mon.noname-c con 0x7fd3580a8e90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd35db6c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3580b2310 0x7fd358138f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd35db6c640 10 monclient(hunting): picked mon.noname-b con 0x7fd3580b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd35db6c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd35db6c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3580ac730 con 0x7fd3580a8ab0 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd35db6c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd35db6c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3580ad940 con 0x7fd3580b2310 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd35db6c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd35db6c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3580adc20 con 0x7fd3580a8e90 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd35db6c640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd35db6c640 10 monclient(hunting): authenticate will time out at 3539.184795s 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd356ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3580a8e90 0x7fd3581389f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd3577fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3580a8ab0 0x7fd3580ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd356ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3580a8e90 0x7fd3581389f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37142/0 (socket says 172.21.15.7:37142) 2024-01-21T03:56:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd356ffd640 1 -- 172.21.15.7:0/277147140 learned_addr learned my addr 172.21.15.7:0/277147140 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd356ffd640 10 monclient(hunting): get_auth_request con 0x7fd3580a8e90 auth_method 0 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd356ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd356ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd357fff640 1 --2- 172.21.15.7:0/277147140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3580b2310 0x7fd358138f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd3577fe640 10 monclient(hunting): get_auth_request con 0x7fd3580a8ab0 auth_method 0 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd3577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd3577fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.869+0000 7fd3577fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.870+0000 7fd357fff640 10 monclient(hunting): get_auth_request con 0x7fd3580b2310 auth_method 0 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.870+0000 7fd357fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.870+0000 7fd357fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.870+0000 7fd357fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.870+0000 7fd356ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:32.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.870+0000 7fd356ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:32.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.870+0000 7fd356ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:32.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.870+0000 7fd3577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:32.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.870+0000 7fd3577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:32.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.870+0000 7fd3577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:32.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.870+0000 7fd357fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:32.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.870+0000 7fd357fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:32.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.870+0000 7fd357fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:32.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.873+0000 7fd356ffd640 10 monclient(hunting): handle_auth_done global_id 18050 payload 824 2024-01-21T03:56:32.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.873+0000 7fd356ffd640 1 -- 172.21.15.7:0/277147140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3580b2310 msgr2=0x7fd358138f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.873+0000 7fd356ffd640 1 --2- 172.21.15.7:0/277147140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3580b2310 0x7fd358138f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.873+0000 7fd356ffd640 1 -- 172.21.15.7:0/277147140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3580a8ab0 msgr2=0x7fd3580ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:32.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.873+0000 7fd356ffd640 1 --2- 172.21.15.7:0/277147140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3580a8ab0 0x7fd3580ac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:32.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.873+0000 7fd356ffd640 10 monclient: _finish_hunting 0 2024-01-21T03:56:32.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.873+0000 7fd356ffd640 1 monclient: found mon.noname-c 2024-01-21T03:56:32.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.873+0000 7fd356ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:32.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.873+0000 7fd356ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:56:32.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.873+0000 7fd356ffd640 1 -- 172.21.15.7:0/277147140 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c009400 con 0x7fd3580a8e90 2024-01-21T03:56:32.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.875+0000 7fd356ffd640 1 --2- 172.21.15.7:0/277147140 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3580a8e90 0x7fd3581389f0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7fd34c034040 tx=0x7fd34c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.875+0000 7fd33ffff640 10 client.?.objecter ms_handle_connect 0x7fd3580a8e90 2024-01-21T03:56:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.875+0000 7fd33ffff640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:32.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.878+0000 7fd33ffff640 1 -- 172.21.15.7:0/277147140 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c032840 con 0x7fd3580a8e90 2024-01-21T03:56:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.878+0000 7fd33ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.878+0000 7fd33ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:56:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.878+0000 7fd33ffff640 10 monclient: dump: 2024-01-21T03:56:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:32.880 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:32.880 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd33ffff640 10 monclient: _finish_auth 0 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd33ffff640 10 monclient: _check_auth_tickets 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd33ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd33ffff640 1 -- 172.21.15.7:0/277147140 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c031f00 con 0x7fd3580a8e90 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd35db6c640 5 monclient: authenticate success, global_id 18050 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd35db6c640 10 monclient: _renew_subs 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd35db6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd35db6c640 1 -- 172.21.15.7:0/277147140 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3580ac9f0 con 0x7fd3580a8e90 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd33ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd33ffff640 1 -- 172.21.15.7:0/277147140 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c032be0 con 0x7fd3580a8e90 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd33ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd33ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd33ffff640 10 monclient: dump: 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:32.881 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.882 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:32.882 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:32.882 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:32.882 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:32.882 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:32.882 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:32.882 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:32.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd35db6c640 10 client.18050.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:32.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd35db6c640 10 monclient: _renew_subs 2024-01-21T03:56:32.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd35db6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:32.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd35db6c640 1 -- 172.21.15.7:0/277147140 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3580ab4e0 con 0x7fd3580a8e90 2024-01-21T03:56:32.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd35db6c640 1 librados: init done 2024-01-21T03:56:32.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd355ffb640 4 set_mon_vals no callback set 2024-01-21T03:56:32.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd35db6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:32.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd35db6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:32.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd35db6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:32.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:32.879+0000 7fd35db6c640 1 -- 172.21.15.7:0/277147140 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35813fc60 con 0x7fd3580a8e90 2024-01-21T03:56:33.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.139+0000 7fd33ffff640 1 -- 172.21.15.7:0/277147140 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd34c032d40 con 0x7fd3580a8e90 2024-01-21T03:56:33.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.140+0000 7fd33ffff640 1 --2- 172.21.15.7:0/277147140 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fd34406c8b0 0x7fd34406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.140+0000 7fd3577fe640 1 --2- 172.21.15.7:0/277147140 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fd34406c8b0 0x7fd34406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.140+0000 7fd3577fe640 10 monclient: get_auth_request con 0x7fd34406c8b0 auth_method 0 2024-01-21T03:56:33.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.140+0000 7fd33ffff640 10 client.18050.objecter ms_handle_connect 0x7fd34406c8b0 2024-01-21T03:56:33.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.140+0000 7fd3577fe640 1 --2- 172.21.15.7:0/277147140 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fd34406c8b0 0x7fd34406ed70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd3480012e0 tx=0x7fd348001a30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:33.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.154+0000 7fd33ffff640 1 -- 172.21.15.7:0/277147140 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7fd34c07a000 con 0x7fd3580a8e90 2024-01-21T03:56:33.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.154+0000 7fd33ffff640 10 client.18050.objecter ms_dispatch 0x7fd358000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:33.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.154+0000 7fd33ffff640 3 client.18050.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:33.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.154+0000 7fd33ffff640 3 client.18050.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:33.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.154+0000 7fd33ffff640 20 client.18050.objecter dump_active .. 0 homeless 2024-01-21T03:56:33.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.154+0000 7fd33ffff640 1 -- 172.21.15.7:0/277147140 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd34c076340 con 0x7fd3580a8e90 2024-01-21T03:56:33.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.154+0000 7fd33ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:33.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.154+0000 7fd33ffff640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:33.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.340+0000 7fd35db6c640 1 -- 172.21.15.7:0/277147140 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd358142190 con 0x7fd34406c8b0 2024-01-21T03:56:33.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.341+0000 7fd33ffff640 1 -- 172.21.15.7:0/277147140 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd348001f60 con 0x7fd34406c8b0 2024-01-21T03:56:33.341 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 4 to deep-scrub 2024-01-21T03:56:33.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.342+0000 7fd35db6c640 10 librados: watch_flush enter 2024-01-21T03:56:33.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.342+0000 7fd35db6c640 10 librados: watch_flush exit 2024-01-21T03:56:33.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.342+0000 7fd35db6c640 20 client.18050.objecter shutdown clearing up homeless session... 2024-01-21T03:56:33.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.342+0000 7fd35db6c640 10 client.18050.objecter successfully canceled tick 2024-01-21T03:56:33.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 1 -- 172.21.15.7:0/277147140 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fd34406c8b0 msgr2=0x7fd34406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 1 --2- 172.21.15.7:0/277147140 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fd34406c8b0 0x7fd34406ed70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd3480012e0 tx=0x7fd348001a30 comp rx=0 tx=0).stop 2024-01-21T03:56:33.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 10 monclient: shutdown 2024-01-21T03:56:33.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:33.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 1 -- 172.21.15.7:0/277147140 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3580a8e90 msgr2=0x7fd3581389f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 1 --2- 172.21.15.7:0/277147140 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3580a8e90 0x7fd3581389f0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7fd34c034040 tx=0x7fd34c0046c0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 1 -- 172.21.15.7:0/277147140 shutdown_connections 2024-01-21T03:56:33.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 1 --2- 172.21.15.7:0/277147140 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fd34406c8b0 0x7fd34406ed70 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 1 --2- 172.21.15.7:0/277147140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3580b2310 0x7fd358138f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 1 --2- 172.21.15.7:0/277147140 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd3580a8e90 0x7fd3581389f0 unknown :-1 s=CLOSED pgs=3158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 1 --2- 172.21.15.7:0/277147140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3580a8ab0 0x7fd3580ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 1 -- 172.21.15.7:0/277147140 >> 172.21.15.7:0/277147140 conn(0x7fd358066f00 msgr2=0x7fd35813cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:33.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 1 -- 172.21.15.7:0/277147140 shutdown_connections 2024-01-21T03:56:33.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 1 -- 172.21.15.7:0/277147140 wait complete. 2024-01-21T03:56:33.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.343+0000 7fd35db6c640 1 librados: shutdown 2024-01-21T03:56:33.354 INFO:tasks.ceph:Scrubbing osd.5 2024-01-21T03:56:33.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-01-21T03:56:33.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.461+0000 7fb0ca52c640 10 monclient: get_monmap_and_config 2024-01-21T03:56:33.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.461+0000 7fb0ca52c640 10 monclient: build_initial_monmap 2024-01-21T03:56:33.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.461+0000 7fb0ca52c640 10 monclient: monmap: 2024-01-21T03:56:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:33.462872+0000 2024-01-21T03:56:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:33.462872+0000 2024-01-21T03:56:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:33.463 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:33.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.462+0000 7fb0ca52c640 1 Processor -- start 2024-01-21T03:56:33.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.462+0000 7fb0ca52c640 1 -- start start 2024-01-21T03:56:33.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.462+0000 7fb0ca52c640 10 monclient: init 2024-01-21T03:56:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-01-21T03:56:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb0c40a8b20 0x7fb0c4145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 10 monclient(hunting): picked mon.noname-c con 0x7fb0c40a8b20 addr v2:172.21.15.177:3300/0 2024-01-21T03:56:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0c40a9d10 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0c40b2310 0x7fb0c413f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 10 monclient(hunting): picked mon.noname-a con 0x7fb0c40b2310 addr v2:172.21.15.7:3300/0 2024-01-21T03:56:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c41456f0 con 0x7fb0c40a9d10 2024-01-21T03:56:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c40a98c0 con 0x7fb0c40b2310 2024-01-21T03:56:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c41460c0 con 0x7fb0c40a8b20 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.463+0000 7fb0ca52c640 10 monclient(hunting): authenticate will time out at 3539.779575s 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.464+0000 7fb0c8ac9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0c40b2310 0x7fb0c413f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.464+0000 7fb0c3fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb0c40a8b20 0x7fb0c4145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.464+0000 7fb0c8ac9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0c40b2310 0x7fb0c413f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44706/0 (socket says 172.21.15.7:44706) 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.464+0000 7fb0c8ac9640 1 -- 172.21.15.7:0/2391928310 learned_addr learned my addr 172.21.15.7:0/2391928310 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.464+0000 7fb0c8ac9640 10 monclient(hunting): get_auth_request con 0x7fb0c40b2310 auth_method 0 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.464+0000 7fb0c8ac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.464+0000 7fb0c8ac9640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.464+0000 7fb0c8ac9640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.465+0000 7fb0c3fff640 10 monclient(hunting): get_auth_request con 0x7fb0c40a8b20 auth_method 0 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.465+0000 7fb0c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.465+0000 7fb0c3fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.465+0000 7fb0c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.465+0000 7fb0c8ac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.465+0000 7fb0c8ac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.465+0000 7fb0c8ac9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.465+0000 7fb0c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.465+0000 7fb0c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.465+0000 7fb0c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.467+0000 7fb0c8ac9640 10 monclient(hunting): handle_auth_done global_id 15891 payload 274 2024-01-21T03:56:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.467+0000 7fb0c8ac9640 1 -- 172.21.15.7:0/2391928310 >> v2:172.21.15.177:3300/0 conn(0x7fb0c40a8b20 msgr2=0x7fb0c4145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.467+0000 7fb0c8ac9640 1 --2- 172.21.15.7:0/2391928310 >> v2:172.21.15.177:3300/0 conn(0x7fb0c40a8b20 0x7fb0c4145b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.467+0000 7fb0c8ac9640 1 -- 172.21.15.7:0/2391928310 >> v1:172.21.15.177:6789/0 conn(0x7fb0c40a9d10 legacy=0x7fb0c413f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.467+0000 7fb0c8ac9640 10 monclient: _finish_hunting 0 2024-01-21T03:56:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.467+0000 7fb0c8ac9640 1 monclient: found mon.noname-a 2024-01-21T03:56:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.467+0000 7fb0c8ac9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.467+0000 7fb0c8ac9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:56:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.467+0000 7fb0c8ac9640 1 -- 172.21.15.7:0/2391928310 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c4146230 con 0x7fb0c40b2310 2024-01-21T03:56:33.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.469+0000 7fb0c8ac9640 1 --2- 172.21.15.7:0/2391928310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0c40b2310 0x7fb0c413f790 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40a8600 tx=0x7fb0bc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9354e07585676522 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0c2ffd640 1 -- 172.21.15.7:0/2391928310 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc02cb20 con 0x7fb0c40b2310 2024-01-21T03:56:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:56:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0c2ffd640 10 monclient: dump: 2024-01-21T03:56:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0c2ffd640 10 monclient: _finish_auth 0 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0c2ffd640 10 monclient: _check_auth_tickets 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0c2ffd640 1 -- 172.21.15.7:0/2391928310 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc02cc80 con 0x7fb0c40b2310 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0ca52c640 5 monclient: authenticate success, global_id 15891 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0ca52c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0ca52c640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0c2ffd640 1 -- 172.21.15.7:0/2391928310 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc0343f0 con 0x7fb0c40b2310 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0ca52c640 4 set_mon_vals no callback set 2024-01-21T03:56:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:56:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.472+0000 7fb0c2ffd640 10 monclient: dump: 2024-01-21T03:56:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:33.475 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:33.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.473+0000 7fb0ca52c640 10 monclient: shutdown 2024-01-21T03:56:33.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.473+0000 7fb0ca52c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.473+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2391928310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0c40b2310 msgr2=0x7fb0c413f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.473+0000 7fb0ca52c640 1 --2- 172.21.15.7:0/2391928310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0c40b2310 0x7fb0c413f790 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40a8600 tx=0x7fb0bc02bf80 comp rx=0 tx=0).stop 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.473+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2391928310 shutdown_connections 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.473+0000 7fb0ca52c640 1 --2- 172.21.15.7:0/2391928310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0c40b2310 0x7fb0c413f790 unknown :-1 s=CLOSED pgs=3095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.473+0000 7fb0ca52c640 1 --2- 172.21.15.7:0/2391928310 >> v2:172.21.15.177:3300/0 conn(0x7fb0c40a8b20 0x7fb0c4145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.473+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2391928310 >> 172.21.15.7:0/2391928310 conn(0x7fb0c4066fb0 msgr2=0x7fb0c40b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.473+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2391928310 shutdown_connections 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.473+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2391928310 wait complete. 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.474+0000 7fb0ca52c640 10 monclient: build_initial_monmap 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.474+0000 7fb0ca52c640 10 monclient: monmap: 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.474+0000 7fb0ca52c640 1 librados: starting msgr at 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.474+0000 7fb0ca52c640 1 librados: starting objecter 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.474+0000 7fb0ca52c640 1 Processor -- start 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.474+0000 7fb0ca52c640 1 -- start start 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.474+0000 7fb0ca52c640 1 librados: setting wanted keys 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.474+0000 7fb0ca52c640 1 librados: calling monclient init 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.474+0000 7fb0ca52c640 10 monclient: init 2024-01-21T03:56:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0c40a8b20 0x7fb0c4142890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 10 monclient(hunting): picked mon.noname-c con 0x7fb0c40a8b20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0c40a9d10 0x7fb0c413fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 10 monclient(hunting): picked mon.noname-a con 0x7fb0c40a9d10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0c40b2310 0x7fb0c41402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 10 monclient(hunting): picked mon.noname-b con 0x7fb0c40b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c4137e60 con 0x7fb0c40a9d10 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c41393a0 con 0x7fb0c40b2310 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c4139680 con 0x7fb0c40a8b20 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0ca52c640 10 monclient(hunting): authenticate will time out at 3539.791106s 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0c37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0c40a9d10 0x7fb0c413fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0c3fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0c40a8b20 0x7fb0c4142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.475+0000 7fb0c8ac9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0c40b2310 0x7fb0c41402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c3fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0c40a8b20 0x7fb0c4142890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37154/0 (socket says 172.21.15.7:37154) 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c3fff640 1 -- 172.21.15.7:0/2267171641 learned_addr learned my addr 172.21.15.7:0/2267171641 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c3fff640 10 monclient(hunting): get_auth_request con 0x7fb0c40a8b20 auth_method 0 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c3fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c37fe640 10 monclient(hunting): get_auth_request con 0x7fb0c40a9d10 auth_method 0 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c37fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c8ac9640 10 monclient(hunting): get_auth_request con 0x7fb0c40b2310 auth_method 0 2024-01-21T03:56:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c8ac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:33.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c8ac9640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:33.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c8ac9640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:33.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:33.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:33.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:33.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:33.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:33.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:33.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c8ac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:33.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c8ac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:33.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.476+0000 7fb0c8ac9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:33.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.479+0000 7fb0c8ac9640 10 monclient(hunting): handle_auth_done global_id 23206 payload 824 2024-01-21T03:56:33.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.479+0000 7fb0c8ac9640 1 -- 172.21.15.7:0/2267171641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0c40a8b20 msgr2=0x7fb0c4142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.479+0000 7fb0c8ac9640 1 --2- 172.21.15.7:0/2267171641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0c40a8b20 0x7fb0c4142890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.479+0000 7fb0c8ac9640 1 -- 172.21.15.7:0/2267171641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0c40a9d10 msgr2=0x7fb0c413fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.479+0000 7fb0c8ac9640 1 --2- 172.21.15.7:0/2267171641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0c40a9d10 0x7fb0c413fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.479+0000 7fb0c8ac9640 10 monclient: _finish_hunting 0 2024-01-21T03:56:33.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.479+0000 7fb0c8ac9640 1 monclient: found mon.noname-b 2024-01-21T03:56:33.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.479+0000 7fb0c8ac9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:33.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.479+0000 7fb0c8ac9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:56:33.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.479+0000 7fb0c8ac9640 1 -- 172.21.15.7:0/2267171641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc009b30 con 0x7fb0c40b2310 2024-01-21T03:56:33.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.481+0000 7fb0c8ac9640 1 --2- 172.21.15.7:0/2267171641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0c40b2310 0x7fb0c41402a0 secure :-1 s=READY pgs=5823 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc032000 tx=0x7fb0bc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:33.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.481+0000 7fb0c0ff9640 10 client.?.objecter ms_handle_connect 0x7fb0c40b2310 2024-01-21T03:56:33.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.481+0000 7fb0c0ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c0ff9640 1 -- 172.21.15.7:0/2267171641 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc02c920 con 0x7fb0c40b2310 2024-01-21T03:56:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c0ff9640 10 monclient: dump: 2024-01-21T03:56:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:33.488 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c0ff9640 10 monclient: _finish_auth 0 2024-01-21T03:56:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c0ff9640 10 monclient: _check_auth_tickets 2024-01-21T03:56:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c0ff9640 1 -- 172.21.15.7:0/2267171641 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc02ca80 con 0x7fb0c40b2310 2024-01-21T03:56:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c0ff9640 1 -- 172.21.15.7:0/2267171641 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc03e690 con 0x7fb0c40b2310 2024-01-21T03:56:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c0ff9640 10 monclient: dump: 2024-01-21T03:56:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0ca52c640 5 monclient: authenticate success, global_id 23206 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0ca52c640 10 monclient: _renew_subs 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0c27fc640 4 set_mon_vals no callback set 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0ca52c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2267171641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c4139960 con 0x7fb0c40b2310 2024-01-21T03:56:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0ca52c640 10 client.23206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0ca52c640 10 monclient: _renew_subs 2024-01-21T03:56:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0ca52c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2267171641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c4142f50 con 0x7fb0c40b2310 2024-01-21T03:56:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.485+0000 7fb0ca52c640 1 librados: init done 2024-01-21T03:56:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.486+0000 7fb0ca52c640 10 client.23206.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.486+0000 7fb0ca52c640 15 client.23206.objecter _session_command_op_assign -1 1 2024-01-21T03:56:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.486+0000 7fb0ca52c640 20 client.23206.objecter _get_session osd=-1 returning homeless 2024-01-21T03:56:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.486+0000 7fb0ca52c640 10 client.23206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.486+0000 7fb0ca52c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.486+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2267171641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb0c40a9110 con 0x7fb0c40b2310 2024-01-21T03:56:33.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.504+0000 7fb0c0ff9640 1 -- 172.21.15.7:0/2267171641 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb0bc03e7f0 con 0x7fb0c40b2310 2024-01-21T03:56:33.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.505+0000 7fb0c0ff9640 1 --2- 172.21.15.7:0/2267171641 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb0b006c8e0 0x7fb0b006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.505+0000 7fb0c3fff640 1 --2- 172.21.15.7:0/2267171641 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb0b006c8e0 0x7fb0b006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.505+0000 7fb0c3fff640 10 monclient: get_auth_request con 0x7fb0b006c8e0 auth_method 0 2024-01-21T03:56:33.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.505+0000 7fb0c3fff640 1 --2- 172.21.15.7:0/2267171641 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb0b006c8e0 0x7fb0b006eda0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009b60 tx=0x7fb0b4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:33.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.505+0000 7fb0c0ff9640 10 client.23206.objecter ms_handle_connect 0x7fb0b006c8e0 2024-01-21T03:56:33.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.507+0000 7fb0c0ff9640 1 -- 172.21.15.7:0/2267171641 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7fb0bc07b2e0 con 0x7fb0c40b2310 2024-01-21T03:56:33.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.507+0000 7fb0c0ff9640 10 client.23206.objecter ms_dispatch 0x7fb0c4000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:33.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.507+0000 7fb0c0ff9640 3 client.23206.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:33.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.507+0000 7fb0c0ff9640 3 client.23206.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:33.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.508+0000 7fb0c0ff9640 10 client.23206.objecter checking command 1 2024-01-21T03:56:33.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.508+0000 7fb0c0ff9640 1 --2- 172.21.15.7:0/2267171641 >> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] conn(0x7fb0b00746d0 0x7fb0b0076b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.508+0000 7fb0c0ff9640 20 client.23206.objecter _get_session s=0x7fb0b0074000 osd=5 3 2024-01-21T03:56:33.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.508+0000 7fb0c0ff9640 20 client.23206.objecter put_session s=0x7fb0b0074000 osd=5 3 2024-01-21T03:56:33.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.508+0000 7fb0c0ff9640 15 client.23206.objecter _session_command_op_remove -1 1 2024-01-21T03:56:33.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.508+0000 7fb0c0ff9640 20 client.23206.objecter _get_session s=0x7fb0b0074000 osd=5 3 2024-01-21T03:56:33.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.508+0000 7fb0c0ff9640 20 client.23206.objecter get_session s=0x7fb0b0074000 osd=5 3 2024-01-21T03:56:33.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.508+0000 7fb0c0ff9640 15 client.23206.objecter _session_command_op_assign 5 1 2024-01-21T03:56:33.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.508+0000 7fb0c0ff9640 20 client.23206.objecter put_session s=0x7fb0b0074000 osd=5 4 2024-01-21T03:56:33.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.508+0000 7fb0c0ff9640 10 client.23206.objecter _send_command 1 2024-01-21T03:56:33.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.508+0000 7fb0c0ff9640 1 -- 172.21.15.7:0/2267171641 --> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb0b0077240 con 0x7fb0b00746d0 2024-01-21T03:56:33.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.508+0000 7fb0c0ff9640 20 client.23206.objecter dump_active .. 0 homeless 2024-01-21T03:56:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.508+0000 7fb0c37fe640 1 --2- 172.21.15.7:0/2267171641 >> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] conn(0x7fb0b00746d0 0x7fb0b0076b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.509+0000 7fb0c0ff9640 1 -- 172.21.15.7:0/2267171641 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=1294) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0bc03ef00 con 0x7fb0c40b2310 2024-01-21T03:56:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.509+0000 7fb0c0ff9640 10 monclient: handle_get_version_reply finishing 1 version 1294 2024-01-21T03:56:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.509+0000 7fb0c37fe640 10 monclient: get_auth_request con 0x7fb0b00746d0 auth_method 0 2024-01-21T03:56:33.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.512+0000 7fb0c37fe640 1 --2- 172.21.15.7:0/2267171641 >> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] conn(0x7fb0b00746d0 0x7fb0b0076b70 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:33.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.512+0000 7fb0c0ff9640 10 client.23206.objecter ms_handle_connect 0x7fb0b00746d0 2024-01-21T03:56:33.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.549+0000 7fb0c0ff9640 1 -- 172.21.15.7:0/2267171641 <== osd.5 v2:172.21.15.177:6816/2318383718 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb0b0077240 con 0x7fb0b00746d0 2024-01-21T03:56:33.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.549+0000 7fb0c0ff9640 10 client.23206.objecter ms_dispatch 0x7fb0c4000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:56:33.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.549+0000 7fb0c0ff9640 10 client.23206.objecter _finish_command 1 = system:0 2024-01-21T03:56:33.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.549+0000 7fb0c0ff9640 20 client.23206.objecter put_session s=0x7fb0b0074000 osd=5 4 2024-01-21T03:56:33.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.549+0000 7fb0c0ff9640 15 client.23206.objecter _session_command_op_remove 5 1 2024-01-21T03:56:33.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.562+0000 7fb0ca52c640 10 client.23206.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T03:56:33.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.562+0000 7fb0ca52c640 15 client.23206.objecter _session_command_op_assign -1 2 2024-01-21T03:56:33.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.562+0000 7fb0ca52c640 20 client.23206.objecter _get_session s=0x7fb0b0074000 osd=5 3 2024-01-21T03:56:33.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.562+0000 7fb0ca52c640 20 client.23206.objecter put_session s=0x7fb0b0074000 osd=5 3 2024-01-21T03:56:33.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.562+0000 7fb0ca52c640 20 client.23206.objecter _get_session s=0x7fb0b0074000 osd=5 3 2024-01-21T03:56:33.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.562+0000 7fb0ca52c640 15 client.23206.objecter _session_command_op_remove -1 2 2024-01-21T03:56:33.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.562+0000 7fb0ca52c640 20 client.23206.objecter get_session s=0x7fb0b0074000 osd=5 3 2024-01-21T03:56:33.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.562+0000 7fb0ca52c640 15 client.23206.objecter _session_command_op_assign 5 2 2024-01-21T03:56:33.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.562+0000 7fb0ca52c640 20 client.23206.objecter put_session s=0x7fb0b0074000 osd=5 4 2024-01-21T03:56:33.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.562+0000 7fb0ca52c640 10 client.23206.objecter _send_command 2 2024-01-21T03:56:33.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.562+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2267171641 --> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb0c41430e0 con 0x7fb0b00746d0 2024-01-21T03:56:33.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0c0ff9640 1 -- 172.21.15.7:0/2267171641 <== osd.5 v2:172.21.15.177:6816/2318383718 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb0c41430e0 con 0x7fb0b00746d0 2024-01-21T03:56:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0c0ff9640 10 client.23206.objecter ms_dispatch 0x7fb0c4000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:56:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0c0ff9640 10 client.23206.objecter _finish_command 2 = system:0 2024-01-21T03:56:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0c0ff9640 20 client.23206.objecter put_session s=0x7fb0b0074000 osd=5 4 2024-01-21T03:56:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0c0ff9640 15 client.23206.objecter _session_command_op_remove 5 2 2024-01-21T03:56:33.576 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-01-21T03:56:33.576 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T03:56:33.576 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-01-21T03:56:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 10 librados: watch_flush enter 2024-01-21T03:56:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 10 librados: watch_flush exit 2024-01-21T03:56:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 10 client.23206.objecter close_session for osd.5 2024-01-21T03:56:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2267171641 >> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] conn(0x7fb0b00746d0 msgr2=0x7fb0b0076b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 1 --2- 172.21.15.7:0/2267171641 >> [v2:172.21.15.177:6816/2318383718,v1:172.21.15.177:6817/2318383718] conn(0x7fb0b00746d0 0x7fb0b0076b70 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 20 client.23206.objecter put_session s=0x7fb0b0074000 osd=5 1 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 20 client.23206.objecter shutdown clearing up homeless session... 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 10 client.23206.objecter successfully canceled tick 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2267171641 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb0b006c8e0 msgr2=0x7fb0b006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 1 --2- 172.21.15.7:0/2267171641 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb0b006c8e0 0x7fb0b006eda0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009b60 tx=0x7fb0b4009190 comp rx=0 tx=0).stop 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 10 monclient: shutdown 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2267171641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0c40b2310 msgr2=0x7fb0c41402a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 1 --2- 172.21.15.7:0/2267171641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0c40b2310 0x7fb0c41402a0 secure :-1 s=READY pgs=5823 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc032000 tx=0x7fb0bc004a40 comp rx=0 tx=0).stop 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0c8ac9640 1 -- 172.21.15.7:0/2267171641 reap_dead start 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2267171641 shutdown_connections 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2267171641 >> 172.21.15.7:0/2267171641 conn(0x7fb0c4066fb0 msgr2=0x7fb0c40b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:33.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2267171641 shutdown_connections 2024-01-21T03:56:33.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.575+0000 7fb0ca52c640 1 -- 172.21.15.7:0/2267171641 wait complete. 2024-01-21T03:56:33.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.576+0000 7fb0ca52c640 1 librados: shutdown 2024-01-21T03:56:33.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-01-21T03:56:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.689+0000 7fad3e573640 10 monclient: get_monmap_and_config 2024-01-21T03:56:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.689+0000 7fad3e573640 10 monclient: build_initial_monmap 2024-01-21T03:56:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.689+0000 7fad3e573640 10 monclient: monmap: 2024-01-21T03:56:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:33.690667+0000 2024-01-21T03:56:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:33.690667+0000 2024-01-21T03:56:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:33.691 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.690+0000 7fad3e573640 1 Processor -- start 2024-01-21T03:56:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.690+0000 7fad3e573640 1 -- start start 2024-01-21T03:56:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.690+0000 7fad3e573640 10 monclient: init 2024-01-21T03:56:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.691+0000 7fad3e573640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.691+0000 7fad3e573640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-01-21T03:56:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.691+0000 7fad3e573640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad380a8a40 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.691+0000 7fad3e573640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380b2210 0x7fad3813f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.691+0000 7fad3e573640 10 monclient(hunting): picked mon.noname-b con 0x7fad380b2210 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.691+0000 7fad3e573640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad380a8e20 addr v1:172.21.15.7:6789/0 2024-01-21T03:56:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.691+0000 7fad3e573640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad38145700 con 0x7fad380a8e20 2024-01-21T03:56:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.691+0000 7fad3e573640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad380aa9b0 con 0x7fad380a8a40 2024-01-21T03:56:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.691+0000 7fad3e573640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.691+0000 7fad3e573640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad380aa7b0 con 0x7fad380b2210 2024-01-21T03:56:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.691+0000 7fad3e573640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.691+0000 7fad3e573640 10 monclient(hunting): authenticate will time out at 3540.006975s 2024-01-21T03:56:33.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.701+0000 7fad37fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fad380a8a40 0x7fad38145b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:42620/0 (socket says 172.21.15.7:42620) 2024-01-21T03:56:33.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.701+0000 7fad37fff640 1 -- 172.21.15.7:0/3293226798 learned_addr learned my addr 172.21.15.7:0/3293226798 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:33.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.706+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755265287 0 0) 0x7fad38145700 con 0x7fad380a8e20 2024-01-21T03:56:33.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.706+0000 7fad36ffd640 10 monclient(hunting): _negotiate 2024-01-21T03:56:33.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.706+0000 7fad36ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:33.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.706+0000 7fad36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:33.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.706+0000 7fad36ffd640 10 monclient(hunting): my global_id is 15897 2024-01-21T03:56:33.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.706+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad20003520 con 0x7fad380a8e20 2024-01-21T03:56:33.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.709+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437130260 0 0) 0x7fad380aa9b0 con 0x7fad380a8a40 2024-01-21T03:56:33.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.709+0000 7fad36ffd640 10 monclient(hunting): _negotiate 2024-01-21T03:56:33.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.709+0000 7fad36ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:33.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.709+0000 7fad36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:33.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.709+0000 7fad36ffd640 10 monclient(hunting): my global_id is 23209 2024-01-21T03:56:33.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.709+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad38145700 con 0x7fad380a8a40 2024-01-21T03:56:33.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad377fe640 1 --2- 172.21.15.7:0/3293226798 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380b2210 0x7fad3813f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1569294151 0 0) 0x7fad38145700 con 0x7fad380a8a40 2024-01-21T03:56:33.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380b2210 msgr2=0x7fad3813f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:33.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 --2- 172.21.15.7:0/3293226798 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380b2210 0x7fad3813f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 >> v1:172.21.15.7:6789/0 conn(0x7fad380a8e20 legacy=0x7fad3814c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient: _finish_hunting 0 2024-01-21T03:56:33.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 monclient: found mon.noname-c-legacy 2024-01-21T03:56:33.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:33.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.177:6789/0 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad380aac90 con 0x7fad380a8a40 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient: _finish_auth 0 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient: _check_auth_tickets 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 749591385 0 0) 0x7fad28001670 con 0x7fad380a8a40 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad3e573640 5 monclient: authenticate success, global_id 23209 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad3e573640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient: dump: 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:33.715 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:33.716 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] but i'm connected to v1:172.21.15.177:6789/0, reconnecting 2024-01-21T03:56:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 >> v1:172.21.15.177:6789/0 conn(0x7fad380a8a40 legacy=0x7fad38145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T03:56:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 --2- 172.21.15.7:0/3293226798 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380b2210 0x7fad20003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient(hunting): picked mon.c con 0x7fad380b2210 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 --2- 172.21.15.7:0/3293226798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad200040e0 0x7fad200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient(hunting): picked mon.a con 0x7fad200040e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:33.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 --2- 172.21.15.7:0/3293226798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad20004e20 0x7fad2000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient(hunting): picked mon.b con 0x7fad20004e20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad380aa7b0 con 0x7fad200040e0 2024-01-21T03:56:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad20005530 con 0x7fad20004e20 2024-01-21T03:56:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2000bec0 con 0x7fad380b2210 2024-01-21T03:56:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad36ffd640 10 monclient(hunting): _finish_auth 0 2024-01-21T03:56:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.712+0000 7fad3e573640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.714+0000 7fad3cb10640 1 --2- 172.21.15.7:0/3293226798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad20004e20 0x7fad2000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.716+0000 7fad3cb10640 10 monclient(hunting): get_auth_request con 0x7fad20004e20 auth_method 0 2024-01-21T03:56:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.716+0000 7fad3cb10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.716+0000 7fad3cb10640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.716+0000 7fad3cb10640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:56:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.716+0000 7fad37fff640 1 --2- 172.21.15.7:0/3293226798 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380b2210 0x7fad20003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.716+0000 7fad37fff640 10 monclient(hunting): get_auth_request con 0x7fad380b2210 auth_method 0 2024-01-21T03:56:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.716+0000 7fad37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.716+0000 7fad37fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.716+0000 7fad37fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T03:56:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.716+0000 7fad3cb10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.716+0000 7fad3cb10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.716+0000 7fad3cb10640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T03:56:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.717+0000 7fad37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.717+0000 7fad37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.717+0000 7fad37fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T03:56:33.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.718+0000 7fad3cb10640 10 monclient(hunting): handle_auth_done global_id 23209 payload 293 2024-01-21T03:56:33.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.718+0000 7fad3cb10640 1 -- 172.21.15.7:0/3293226798 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380b2210 msgr2=0x7fad20003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:33.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.718+0000 7fad3cb10640 1 --2- 172.21.15.7:0/3293226798 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380b2210 0x7fad20003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.718+0000 7fad3cb10640 1 -- 172.21.15.7:0/3293226798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad200040e0 msgr2=0x7fad200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:33.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.718+0000 7fad3cb10640 1 --2- 172.21.15.7:0/3293226798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad200040e0 0x7fad200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.718+0000 7fad3cb10640 10 monclient: _finish_hunting 0 2024-01-21T03:56:33.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.718+0000 7fad3cb10640 1 monclient: found mon.b 2024-01-21T03:56:33.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.718+0000 7fad3cb10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:33.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.718+0000 7fad3cb10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:33.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.718+0000 7fad3cb10640 1 -- 172.21.15.7:0/3293226798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad38145700 con 0x7fad20004e20 2024-01-21T03:56:33.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.718+0000 7fad3cb10640 10 monclient: _finish_auth 0 2024-01-21T03:56:33.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.718+0000 7fad3cb10640 10 monclient: _check_auth_tickets 2024-01-21T03:56:33.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.718+0000 7fad3cb10640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:33.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.721+0000 7fad3cb10640 1 --2- 172.21.15.7:0/3293226798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad20004e20 0x7fad2000b980 secure :-1 s=READY pgs=5825 cs=0 l=1 rev1=1 crypto rx=0x7fad30001fd0 tx=0x7fad200057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c8affd0811654c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.724+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad30003b70 con 0x7fad20004e20 2024-01-21T03:56:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.724+0000 7fad36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.724+0000 7fad36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.724+0000 7fad36ffd640 10 monclient: dump: 2024-01-21T03:56:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:33.725 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.724+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad30003cf0 con 0x7fad20004e20 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.724+0000 7fad36ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.724+0000 7fad36ffd640 1 -- 172.21.15.7:0/3293226798 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad30005310 con 0x7fad20004e20 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.724+0000 7fad3e573640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.724+0000 7fad3e573640 4 set_mon_vals no callback set 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.724+0000 7fad36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.724+0000 7fad36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.724+0000 7fad36ffd640 10 monclient: dump: 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:33.726 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:33.727 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:33.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.725+0000 7fad3e573640 10 monclient: shutdown 2024-01-21T03:56:33.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.725+0000 7fad3e573640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:33.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.725+0000 7fad3e573640 1 -- 172.21.15.7:0/3293226798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad20004e20 msgr2=0x7fad2000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.725+0000 7fad3e573640 1 --2- 172.21.15.7:0/3293226798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad20004e20 0x7fad2000b980 secure :-1 s=READY pgs=5825 cs=0 l=1 rev1=1 crypto rx=0x7fad30001fd0 tx=0x7fad200057f0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.725+0000 7fad3cb10640 1 -- 172.21.15.7:0/3293226798 reap_dead start 2024-01-21T03:56:33.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.725+0000 7fad3e573640 1 -- 172.21.15.7:0/3293226798 shutdown_connections 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.725+0000 7fad3e573640 1 -- 172.21.15.7:0/3293226798 >> 172.21.15.7:0/3293226798 conn(0x7fad38066f10 msgr2=0x7fad380b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.725+0000 7fad3e573640 1 -- 172.21.15.7:0/3293226798 shutdown_connections 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.725+0000 7fad3e573640 1 -- 172.21.15.7:0/3293226798 wait complete. 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.726+0000 7fad3e573640 10 monclient: build_initial_monmap 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.726+0000 7fad3e573640 10 monclient: monmap: 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.726+0000 7fad3e573640 1 librados: starting msgr at 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.726+0000 7fad3e573640 1 librados: starting objecter 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.726+0000 7fad3e573640 1 Processor -- start 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.726+0000 7fad3e573640 1 -- start start 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.726+0000 7fad3e573640 1 librados: setting wanted keys 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.726+0000 7fad3e573640 1 librados: calling monclient init 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.726+0000 7fad3e573640 10 monclient: init 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad20004e20 0x7fad380adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 10 monclient(hunting): picked mon.noname-a con 0x7fad20004e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380aea80 0x7fad381d7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 10 monclient(hunting): picked mon.noname-c con 0x7fad380aea80 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad381d7a80 0x7fad380b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 10 monclient(hunting): picked mon.noname-b con 0x7fad381d7a80 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad380afba0 con 0x7fad20004e20 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381d70b0 con 0x7fad381d7a80 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad380aeec0 con 0x7fad380aea80 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad3e573640 10 monclient(hunting): authenticate will time out at 3540.043268s 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad377fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380aea80 0x7fad381d7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.727+0000 7fad37fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad20004e20 0x7fad380adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad3cb10640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad381d7a80 0x7fad380b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad377fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380aea80 0x7fad381d7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37182/0 (socket says 172.21.15.7:37182) 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad377fe640 1 -- 172.21.15.7:0/971626977 learned_addr learned my addr 172.21.15.7:0/971626977 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:33.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad377fe640 10 monclient(hunting): get_auth_request con 0x7fad380aea80 auth_method 0 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad377fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad377fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad37fff640 10 monclient(hunting): get_auth_request con 0x7fad20004e20 auth_method 0 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad37fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad37fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad3cb10640 10 monclient(hunting): get_auth_request con 0x7fad381d7a80 auth_method 0 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad3cb10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad3cb10640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad3cb10640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad3cb10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad3cb10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad3cb10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:33.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:33.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:33.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:33.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:33.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.728+0000 7fad37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:33.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.731+0000 7fad3cb10640 10 monclient(hunting): handle_auth_done global_id 23212 payload 824 2024-01-21T03:56:33.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.731+0000 7fad3cb10640 1 -- 172.21.15.7:0/971626977 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380aea80 msgr2=0x7fad381d7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.731+0000 7fad3cb10640 1 --2- 172.21.15.7:0/971626977 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380aea80 0x7fad381d7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.731+0000 7fad3cb10640 1 -- 172.21.15.7:0/971626977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad20004e20 msgr2=0x7fad380adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:33.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.731+0000 7fad3cb10640 1 --2- 172.21.15.7:0/971626977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad20004e20 0x7fad380adce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:33.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.731+0000 7fad3cb10640 10 monclient: _finish_hunting 0 2024-01-21T03:56:33.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.731+0000 7fad3cb10640 1 monclient: found mon.noname-b 2024-01-21T03:56:33.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.731+0000 7fad3cb10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:33.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.731+0000 7fad3cb10640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:56:33.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.731+0000 7fad3cb10640 1 -- 172.21.15.7:0/971626977 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad381d7f00 con 0x7fad381d7a80 2024-01-21T03:56:33.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.733+0000 7fad3cb10640 1 --2- 172.21.15.7:0/971626977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad381d7a80 0x7fad380b0ad0 secure :-1 s=READY pgs=5826 cs=0 l=1 rev1=1 crypto rx=0x7fad30004000 tx=0x7fad30001650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:33.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.733+0000 7fad34ff9640 10 client.?.objecter ms_handle_connect 0x7fad381d7a80 2024-01-21T03:56:33.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.733+0000 7fad34ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad34ff9640 1 -- 172.21.15.7:0/971626977 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad30003e60 con 0x7fad381d7a80 2024-01-21T03:56:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad34ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad34ff9640 10 monclient: dump: 2024-01-21T03:56:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad34ff9640 10 monclient: _finish_auth 0 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad34ff9640 10 monclient: _check_auth_tickets 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad34ff9640 1 -- 172.21.15.7:0/971626977 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad30007db0 con 0x7fad381d7a80 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad34ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad34ff9640 1 -- 172.21.15.7:0/971626977 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad300057e0 con 0x7fad381d7a80 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad34ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad34ff9640 10 monclient: dump: 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad3e573640 5 monclient: authenticate success, global_id 23212 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad3e573640 10 monclient: _renew_subs 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad3e573640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad3e573640 1 -- 172.21.15.7:0/971626977 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad380afdf0 con 0x7fad381d7a80 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad367fc640 4 set_mon_vals no callback set 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad3e573640 10 client.23212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad3e573640 10 monclient: _renew_subs 2024-01-21T03:56:33.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad3e573640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:33.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad3e573640 1 -- 172.21.15.7:0/971626977 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad380b04f0 con 0x7fad381d7a80 2024-01-21T03:56:33.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.737+0000 7fad3e573640 1 librados: init done 2024-01-21T03:56:33.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.738+0000 7fad3e573640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:33.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.738+0000 7fad3e573640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:33.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.738+0000 7fad3e573640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:33.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.738+0000 7fad3e573640 1 -- 172.21.15.7:0/971626977 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad380b0680 con 0x7fad381d7a80 2024-01-21T03:56:33.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.979+0000 7fad34ff9640 1 -- 172.21.15.7:0/971626977 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fad3002c550 con 0x7fad381d7a80 2024-01-21T03:56:33.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.979+0000 7fad34ff9640 1 --2- 172.21.15.7:0/971626977 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fad2406c860 0x7fad2406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:33.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.979+0000 7fad37fff640 1 --2- 172.21.15.7:0/971626977 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fad2406c860 0x7fad2406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:33.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.979+0000 7fad37fff640 10 monclient: get_auth_request con 0x7fad2406c860 auth_method 0 2024-01-21T03:56:33.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.980+0000 7fad34ff9640 10 client.23212.objecter ms_handle_connect 0x7fad2406c860 2024-01-21T03:56:33.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.980+0000 7fad37fff640 1 --2- 172.21.15.7:0/971626977 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fad2406c860 0x7fad2406ed20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fad28000fb0 tx=0x7fad28001b30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:33.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.981+0000 7fad34ff9640 1 -- 172.21.15.7:0/971626977 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7fad30079ce0 con 0x7fad381d7a80 2024-01-21T03:56:33.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.981+0000 7fad34ff9640 10 client.23212.objecter ms_dispatch 0x7fad38000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:33.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.981+0000 7fad34ff9640 3 client.23212.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:33.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.981+0000 7fad34ff9640 3 client.23212.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:33.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.981+0000 7fad34ff9640 20 client.23212.objecter dump_active .. 0 homeless 2024-01-21T03:56:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.992+0000 7fad34ff9640 1 -- 172.21.15.7:0/971626977 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fad30076020 con 0x7fad381d7a80 2024-01-21T03:56:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.992+0000 7fad34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:33.992+0000 7fad34ff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.178+0000 7fad3e573640 1 -- 172.21.15.7:0/971626977 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fad380b0680 con 0x7fad2406c860 2024-01-21T03:56:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.178+0000 7fad34ff9640 1 -- 172.21.15.7:0/971626977 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fad380b0680 con 0x7fad2406c860 2024-01-21T03:56:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 5 to deep-scrub 2024-01-21T03:56:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 10 librados: watch_flush enter 2024-01-21T03:56:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 10 librados: watch_flush exit 2024-01-21T03:56:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 20 client.23212.objecter shutdown clearing up homeless session... 2024-01-21T03:56:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 10 client.23212.objecter successfully canceled tick 2024-01-21T03:56:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 1 -- 172.21.15.7:0/971626977 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fad2406c860 msgr2=0x7fad2406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 1 --2- 172.21.15.7:0/971626977 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fad2406c860 0x7fad2406ed20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fad28000fb0 tx=0x7fad28001b30 comp rx=0 tx=0).stop 2024-01-21T03:56:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 10 monclient: shutdown 2024-01-21T03:56:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 1 -- 172.21.15.7:0/971626977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad381d7a80 msgr2=0x7fad380b0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 1 --2- 172.21.15.7:0/971626977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad381d7a80 0x7fad380b0ad0 secure :-1 s=READY pgs=5826 cs=0 l=1 rev1=1 crypto rx=0x7fad30004000 tx=0x7fad30001650 comp rx=0 tx=0).stop 2024-01-21T03:56:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 1 -- 172.21.15.7:0/971626977 shutdown_connections 2024-01-21T03:56:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 1 --2- 172.21.15.7:0/971626977 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fad2406c860 0x7fad2406ed20 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 1 --2- 172.21.15.7:0/971626977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad381d7a80 0x7fad380b0ad0 unknown :-1 s=CLOSED pgs=5826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 1 --2- 172.21.15.7:0/971626977 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fad380aea80 0x7fad381d7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 1 --2- 172.21.15.7:0/971626977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad20004e20 0x7fad380adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 1 -- 172.21.15.7:0/971626977 >> 172.21.15.7:0/971626977 conn(0x7fad38066f10 msgr2=0x7fad3813ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 1 -- 172.21.15.7:0/971626977 shutdown_connections 2024-01-21T03:56:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 1 -- 172.21.15.7:0/971626977 wait complete. 2024-01-21T03:56:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.180+0000 7fad3e573640 1 librados: shutdown 2024-01-21T03:56:34.192 INFO:tasks.ceph:Scrubbing osd.6 2024-01-21T03:56:34.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-01-21T03:56:34.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.308+0000 7f6ed1875640 10 monclient: get_monmap_and_config 2024-01-21T03:56:34.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.308+0000 7f6ed1875640 10 monclient: build_initial_monmap 2024-01-21T03:56:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.308+0000 7f6ed1875640 10 monclient: monmap: 2024-01-21T03:56:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:34.309102+0000 2024-01-21T03:56:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:34.309102+0000 2024-01-21T03:56:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:34.309 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.308+0000 7f6ed1875640 1 Processor -- start 2024-01-21T03:56:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.308+0000 7f6ed1875640 1 -- start start 2024-01-21T03:56:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.308+0000 7f6ed1875640 10 monclient: init 2024-01-21T03:56:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-01-21T03:56:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ecc0a8b20 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ecc0a9d10 0x7f6ecc13f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 10 monclient(hunting): picked mon.noname-b con 0x7f6ecc0a9d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6ecc0b2310 0x7f6ecc13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 10 monclient(hunting): picked mon.noname-c con 0x7f6ecc0b2310 addr v2:172.21.15.177:3300/0 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc1456f0 con 0x7f6ecc0a8b20 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ecc0a98c0 con 0x7f6ecc0b2310 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc13fb00 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6ed1875640 10 monclient(hunting): authenticate will time out at 3540.625069s 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.309+0000 7f6eca7fc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ecc0a9d10 0x7f6ecc13f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.310+0000 7f6ecb7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6ecc0b2310 0x7f6ecc13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.310+0000 7f6eca7fc640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ecc0a9d10 0x7f6ecc13f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37194/0 (socket says 172.21.15.7:37194) 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.310+0000 7f6eca7fc640 1 -- 172.21.15.7:0/143889790 learned_addr learned my addr 172.21.15.7:0/143889790 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.310+0000 7f6eca7fc640 10 monclient(hunting): get_auth_request con 0x7f6ecc0a9d10 auth_method 0 2024-01-21T03:56:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.310+0000 7f6eca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.310+0000 7f6eca7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.310+0000 7f6eca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.310+0000 7f6eca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.310+0000 7f6eca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.310+0000 7f6eca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.311+0000 7f6ecb7fe640 10 monclient(hunting): get_auth_request con 0x7f6ecc0b2310 auth_method 0 2024-01-21T03:56:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.311+0000 7f6ecb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.311+0000 7f6ecb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.311+0000 7f6ecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.312+0000 7f6eca7fc640 10 monclient(hunting): handle_auth_done global_id 18059 payload 274 2024-01-21T03:56:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.312+0000 7f6eca7fc640 1 -- 172.21.15.7:0/143889790 >> v2:172.21.15.177:3300/0 conn(0x7f6ecc0b2310 msgr2=0x7f6ecc13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.312+0000 7f6eca7fc640 1 --2- 172.21.15.7:0/143889790 >> v2:172.21.15.177:3300/0 conn(0x7f6ecc0b2310 0x7f6ecc13f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.312+0000 7f6eca7fc640 1 -- 172.21.15.7:0/143889790 >> v1:172.21.15.177:6789/0 conn(0x7f6ecc0a8b20 legacy=0x7f6ecc145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.312+0000 7f6eca7fc640 10 monclient: _finish_hunting 0 2024-01-21T03:56:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.312+0000 7f6eca7fc640 1 monclient: found mon.noname-b 2024-01-21T03:56:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.312+0000 7f6eca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.312+0000 7f6eca7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:56:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.312+0000 7f6eca7fc640 1 -- 172.21.15.7:0/143889790 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc146290 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.314+0000 7f6eca7fc640 1 --2- 172.21.15.7:0/143889790 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ecc0a9d10 0x7f6ecc13f080 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0009870 tx=0x7f6ec002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dda948dfb360882 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ec9ffb640 1 -- 172.21.15.7:0/143889790 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0030840 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ec9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ec9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ec9ffb640 10 monclient: dump: 2024-01-21T03:56:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ec9ffb640 10 monclient: _finish_auth 0 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ec9ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ec9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ec9ffb640 1 -- 172.21.15.7:0/143889790 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec00309a0 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 5 monclient: authenticate success, global_id 18059 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ec9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 4 set_mon_vals no callback set 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ec9ffb640 1 -- 172.21.15.7:0/143889790 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0030d30 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ec9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ec9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:34.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ec9ffb640 10 monclient: dump: 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 10 monclient: shutdown 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 1 -- 172.21.15.7:0/143889790 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ecc0a9d10 msgr2=0x7f6ecc13f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 1 --2- 172.21.15.7:0/143889790 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ecc0a9d10 0x7f6ecc13f080 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0009870 tx=0x7f6ec002b980 comp rx=0 tx=0).stop 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 1 -- 172.21.15.7:0/143889790 shutdown_connections 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 1 --2- 172.21.15.7:0/143889790 >> v2:172.21.15.177:3300/0 conn(0x7f6ecc0b2310 0x7f6ecc13f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 1 --2- 172.21.15.7:0/143889790 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ecc0a9d10 0x7f6ecc13f080 unknown :-1 s=CLOSED pgs=3159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 1 -- 172.21.15.7:0/143889790 >> 172.21.15.7:0/143889790 conn(0x7f6ecc066fb0 msgr2=0x7f6ecc0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 1 -- 172.21.15.7:0/143889790 shutdown_connections 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.318+0000 7f6ed1875640 1 -- 172.21.15.7:0/143889790 wait complete. 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.319+0000 7f6ed1875640 10 monclient: build_initial_monmap 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.319+0000 7f6ed1875640 10 monclient: monmap: 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:34.323 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:34.324 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:34.324 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:34.324 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:34.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.319+0000 7f6ed1875640 1 librados: starting msgr at 2024-01-21T03:56:34.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.319+0000 7f6ed1875640 1 librados: starting objecter 2024-01-21T03:56:34.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.319+0000 7f6ed1875640 1 Processor -- start 2024-01-21T03:56:34.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 1 -- start start 2024-01-21T03:56:34.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 1 librados: setting wanted keys 2024-01-21T03:56:34.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 1 librados: calling monclient init 2024-01-21T03:56:34.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 10 monclient: init 2024-01-21T03:56:34.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:34.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T03:56:34.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ecc0a8b20 0x7f6ecc1da860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:34.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 10 monclient(hunting): picked mon.noname-c con 0x7f6ecc0a8b20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ecc0a9d10 0x7f6ecc140590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 10 monclient(hunting): picked mon.noname-a con 0x7f6ecc0a9d10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ecc0b2310 0x7f6ecc141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 10 monclient(hunting): picked mon.noname-b con 0x7f6ecc0b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc13ff20 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc140bf0 con 0x7f6ecc0b2310 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc139da0 con 0x7f6ecc0a8b20 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.320+0000 7f6ed1875640 10 monclient(hunting): authenticate will time out at 3540.636469s 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.321+0000 7f6eca7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ecc0a9d10 0x7f6ecc140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:34.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.321+0000 7f6eca7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ecc0a9d10 0x7f6ecc140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44742/0 (socket says 172.21.15.7:44742) 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.321+0000 7f6eca7fc640 1 -- 172.21.15.7:0/2676368047 learned_addr learned my addr 172.21.15.7:0/2676368047 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.321+0000 7f6eca7fc640 10 monclient(hunting): get_auth_request con 0x7f6ecc0a9d10 auth_method 0 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.321+0000 7f6eca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.321+0000 7f6eca7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.321+0000 7f6eca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.321+0000 7f6ecb7fe640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ecc0b2310 0x7f6ecc141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.321+0000 7f6ecaffd640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ecc0a8b20 0x7f6ecc1da860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecb7fe640 10 monclient(hunting): get_auth_request con 0x7f6ecc0b2310 auth_method 0 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecaffd640 10 monclient(hunting): get_auth_request con 0x7f6ecc0a8b20 auth_method 0 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecaffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6eca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6eca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6eca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.322+0000 7f6ecaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.325+0000 7f6eca7fc640 10 monclient(hunting): handle_auth_done global_id 15903 payload 824 2024-01-21T03:56:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.325+0000 7f6eca7fc640 1 -- 172.21.15.7:0/2676368047 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ecc0a8b20 msgr2=0x7f6ecc1da860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.325+0000 7f6eca7fc640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ecc0a8b20 0x7f6ecc1da860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.325+0000 7f6eca7fc640 1 -- 172.21.15.7:0/2676368047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ecc0b2310 msgr2=0x7f6ecc141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.325+0000 7f6eca7fc640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ecc0b2310 0x7f6ecc141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:34.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.325+0000 7f6eca7fc640 10 monclient: _finish_hunting 0 2024-01-21T03:56:34.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.325+0000 7f6eca7fc640 1 monclient: found mon.noname-a 2024-01-21T03:56:34.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.325+0000 7f6eca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:34.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.325+0000 7f6eca7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:56:34.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.325+0000 7f6eca7fc640 1 -- 172.21.15.7:0/2676368047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec00094d0 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.327+0000 7f6eca7fc640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ecc0a9d10 0x7f6ecc140590 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0000c00 tx=0x7f6ec0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.327+0000 7f6eb3fff640 10 client.?.objecter ms_handle_connect 0x7f6ecc0a9d10 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.327+0000 7f6eb3fff640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6eb3fff640 1 -- 172.21.15.7:0/2676368047 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0039c90 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6eb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6eb3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6eb3fff640 10 monclient: dump: 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:34.333 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6eb3fff640 10 monclient: _finish_auth 0 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6eb3fff640 10 monclient: _check_auth_tickets 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6eb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6eb3fff640 1 -- 172.21.15.7:0/2676368047 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec0039df0 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6eb3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6eb3fff640 1 -- 172.21.15.7:0/2676368047 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0043640 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6ed1875640 5 monclient: authenticate success, global_id 15903 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6ed1875640 10 monclient: _renew_subs 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6ed1875640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6ed1875640 1 -- 172.21.15.7:0/2676368047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ecc1db2a0 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6eb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6eb3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6eb3fff640 10 monclient: dump: 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6ec97fa640 4 set_mon_vals no callback set 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6ed1875640 10 client.15903.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6ed1875640 10 monclient: _renew_subs 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6ed1875640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6ed1875640 1 -- 172.21.15.7:0/2676368047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ecc141830 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.331+0000 7f6ed1875640 1 librados: init done 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.332+0000 7f6ed1875640 10 client.15903.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:34.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.332+0000 7f6ed1875640 15 client.15903.objecter _session_command_op_assign -1 1 2024-01-21T03:56:34.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.332+0000 7f6ed1875640 20 client.15903.objecter _get_session osd=-1 returning homeless 2024-01-21T03:56:34.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.332+0000 7f6ed1875640 10 client.15903.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:34.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.332+0000 7f6ed1875640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-01-21T03:56:34.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.332+0000 7f6ed1875640 1 -- 172.21.15.7:0/2676368047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ecc0a9110 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.351+0000 7f6eb3fff640 1 -- 172.21.15.7:0/2676368047 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6ec00437a0 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.351+0000 7f6eb3fff640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6eb806c8b0 0x7f6eb806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:34.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.351+0000 7f6ecaffd640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6eb806c8b0 0x7f6eb806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:34.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.351+0000 7f6ecaffd640 10 monclient: get_auth_request con 0x7f6eb806c8b0 auth_method 0 2024-01-21T03:56:34.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.352+0000 7f6eb3fff640 10 client.15903.objecter ms_handle_connect 0x7f6eb806c8b0 2024-01-21T03:56:34.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.352+0000 7f6ecaffd640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6eb806c8b0 0x7f6eb806ed70 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc001e70 tx=0x7f6ebc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:34.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 1 -- 172.21.15.7:0/2676368047 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7f6ec003e070 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 10 client.15903.objecter ms_dispatch 0x7f6ecc000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:34.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 3 client.15903.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:34.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 3 client.15903.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:34.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 10 client.15903.objecter checking command 1 2024-01-21T03:56:34.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] conn(0x7f6eb80746a0 0x7f6eb8076b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 20 client.15903.objecter _get_session s=0x7f6eb8073fd0 osd=6 3 2024-01-21T03:56:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 20 client.15903.objecter put_session s=0x7f6eb8073fd0 osd=6 3 2024-01-21T03:56:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 15 client.15903.objecter _session_command_op_remove -1 1 2024-01-21T03:56:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 20 client.15903.objecter _get_session s=0x7f6eb8073fd0 osd=6 3 2024-01-21T03:56:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 20 client.15903.objecter get_session s=0x7f6eb8073fd0 osd=6 3 2024-01-21T03:56:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 15 client.15903.objecter _session_command_op_assign 6 1 2024-01-21T03:56:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 20 client.15903.objecter put_session s=0x7f6eb8073fd0 osd=6 4 2024-01-21T03:56:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 10 client.15903.objecter _send_command 1 2024-01-21T03:56:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 1 -- 172.21.15.7:0/2676368047 --> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6eb8077210 con 0x7f6eb80746a0 2024-01-21T03:56:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6eb3fff640 20 client.15903.objecter dump_active .. 0 homeless 2024-01-21T03:56:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.355+0000 7f6ecb7fe640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] conn(0x7f6eb80746a0 0x7f6eb8076b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.356+0000 7f6ecb7fe640 10 monclient: get_auth_request con 0x7f6eb80746a0 auth_method 0 2024-01-21T03:56:34.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.357+0000 7f6eb3fff640 1 -- 172.21.15.7:0/2676368047 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=1294) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ec0078010 con 0x7f6ecc0a9d10 2024-01-21T03:56:34.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.357+0000 7f6eb3fff640 10 monclient: handle_get_version_reply finishing 1 version 1294 2024-01-21T03:56:34.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.358+0000 7f6eb3fff640 10 client.15903.objecter ms_handle_connect 0x7f6eb80746a0 2024-01-21T03:56:34.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.358+0000 7f6ecb7fe640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] conn(0x7f6eb80746a0 0x7f6eb8076b40 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:34.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.383+0000 7f6eb3fff640 1 -- 172.21.15.7:0/2676368047 <== osd.6 v2:172.21.15.177:6800/1092304531 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6eb8077210 con 0x7f6eb80746a0 2024-01-21T03:56:34.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.383+0000 7f6eb3fff640 10 client.15903.objecter ms_dispatch 0x7f6ecc000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:56:34.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.383+0000 7f6eb3fff640 10 client.15903.objecter _finish_command 1 = system:0 2024-01-21T03:56:34.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.383+0000 7f6eb3fff640 20 client.15903.objecter put_session s=0x7f6eb8073fd0 osd=6 4 2024-01-21T03:56:34.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.383+0000 7f6eb3fff640 15 client.15903.objecter _session_command_op_remove 6 1 2024-01-21T03:56:34.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.396+0000 7f6ed1875640 10 client.15903.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T03:56:34.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.396+0000 7f6ed1875640 15 client.15903.objecter _session_command_op_assign -1 2 2024-01-21T03:56:34.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.396+0000 7f6ed1875640 20 client.15903.objecter _get_session s=0x7f6eb8073fd0 osd=6 3 2024-01-21T03:56:34.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.396+0000 7f6ed1875640 20 client.15903.objecter put_session s=0x7f6eb8073fd0 osd=6 3 2024-01-21T03:56:34.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.396+0000 7f6ed1875640 20 client.15903.objecter _get_session s=0x7f6eb8073fd0 osd=6 3 2024-01-21T03:56:34.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.396+0000 7f6ed1875640 15 client.15903.objecter _session_command_op_remove -1 2 2024-01-21T03:56:34.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.396+0000 7f6ed1875640 20 client.15903.objecter get_session s=0x7f6eb8073fd0 osd=6 3 2024-01-21T03:56:34.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.396+0000 7f6ed1875640 15 client.15903.objecter _session_command_op_assign 6 2 2024-01-21T03:56:34.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.396+0000 7f6ed1875640 20 client.15903.objecter put_session s=0x7f6eb8073fd0 osd=6 4 2024-01-21T03:56:34.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.396+0000 7f6ed1875640 10 client.15903.objecter _send_command 2 2024-01-21T03:56:34.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.396+0000 7f6ed1875640 1 -- 172.21.15.7:0/2676368047 --> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6ecc1db570 con 0x7f6eb80746a0 2024-01-21T03:56:34.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.411+0000 7f6eb3fff640 1 -- 172.21.15.7:0/2676368047 <== osd.6 v2:172.21.15.177:6800/1092304531 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6ecc1db570 con 0x7f6eb80746a0 2024-01-21T03:56:34.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.411+0000 7f6eb3fff640 10 client.15903.objecter ms_dispatch 0x7f6ecc000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:56:34.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.411+0000 7f6eb3fff640 10 client.15903.objecter _finish_command 2 = system:0 2024-01-21T03:56:34.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.411+0000 7f6eb3fff640 20 client.15903.objecter put_session s=0x7f6eb8073fd0 osd=6 4 2024-01-21T03:56:34.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.411+0000 7f6eb3fff640 15 client.15903.objecter _session_command_op_remove 6 2 2024-01-21T03:56:34.412 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-01-21T03:56:34.413 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T03:56:34.413 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-01-21T03:56:34.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 10 librados: watch_flush enter 2024-01-21T03:56:34.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 10 librados: watch_flush exit 2024-01-21T03:56:34.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 10 client.15903.objecter close_session for osd.6 2024-01-21T03:56:34.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 1 -- 172.21.15.7:0/2676368047 >> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] conn(0x7f6eb80746a0 msgr2=0x7f6eb8076b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.177:6800/1092304531,v1:172.21.15.177:6801/1092304531] conn(0x7f6eb80746a0 0x7f6eb8076b40 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:34.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 20 client.15903.objecter put_session s=0x7f6eb8073fd0 osd=6 1 2024-01-21T03:56:34.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 20 client.15903.objecter shutdown clearing up homeless session... 2024-01-21T03:56:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 10 client.15903.objecter successfully canceled tick 2024-01-21T03:56:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 1 -- 172.21.15.7:0/2676368047 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6eb806c8b0 msgr2=0x7f6eb806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f6eb806c8b0 0x7f6eb806ed70 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc001e70 tx=0x7f6ebc009040 comp rx=0 tx=0).stop 2024-01-21T03:56:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 10 monclient: shutdown 2024-01-21T03:56:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 1 -- 172.21.15.7:0/2676368047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ecc0a9d10 msgr2=0x7f6ecc140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 1 --2- 172.21.15.7:0/2676368047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ecc0a9d10 0x7f6ecc140590 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0000c00 tx=0x7f6ec0004730 comp rx=0 tx=0).stop 2024-01-21T03:56:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ecb7fe640 1 -- 172.21.15.7:0/2676368047 reap_dead start 2024-01-21T03:56:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 1 -- 172.21.15.7:0/2676368047 shutdown_connections 2024-01-21T03:56:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 1 -- 172.21.15.7:0/2676368047 >> 172.21.15.7:0/2676368047 conn(0x7f6ecc066fb0 msgr2=0x7f6ecc0b33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 1 -- 172.21.15.7:0/2676368047 shutdown_connections 2024-01-21T03:56:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 1 -- 172.21.15.7:0/2676368047 wait complete. 2024-01-21T03:56:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.412+0000 7f6ed1875640 1 librados: shutdown 2024-01-21T03:56:34.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-01-21T03:56:34.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.525+0000 7fda7a234640 10 monclient: get_monmap_and_config 2024-01-21T03:56:34.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.525+0000 7fda7a234640 10 monclient: build_initial_monmap 2024-01-21T03:56:34.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.525+0000 7fda7a234640 10 monclient: monmap: 2024-01-21T03:56:34.526 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:34.526 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:34.526 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:34.526706+0000 2024-01-21T03:56:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:34.526706+0000 2024-01-21T03:56:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:34.527 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.526+0000 7fda7a234640 1 Processor -- start 2024-01-21T03:56:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.526+0000 7fda7a234640 1 -- start start 2024-01-21T03:56:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.526+0000 7fda7a234640 10 monclient: init 2024-01-21T03:56:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda7a234640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda7a234640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-01-21T03:56:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda7a234640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda740a8b20 addr v1:172.21.15.7:6789/0 2024-01-21T03:56:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda7a234640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fda740a9d10 0x7fda7413f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda7a234640 10 monclient(hunting): picked mon.noname-a con 0x7fda740a9d10 addr v2:172.21.15.7:3300/0 2024-01-21T03:56:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda7a234640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda740b2310 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda7a234640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda741456f0 con 0x7fda740a8b20 2024-01-21T03:56:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda7a234640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda740a98c0 con 0x7fda740b2310 2024-01-21T03:56:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda7a234640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:34.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda7a234640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda74146290 con 0x7fda740a9d10 2024-01-21T03:56:34.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda7a234640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:34.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda7a234640 10 monclient(hunting): authenticate will time out at 3540.842861s 2024-01-21T03:56:34.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda72ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fda740a9d10 0x7fda7413f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:34.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda73fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fda740b2310 0x7fda7413f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:42646/0 (socket says 172.21.15.7:42646) 2024-01-21T03:56:34.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.527+0000 7fda73fff640 1 -- 172.21.15.7:0/2538200880 learned_addr learned my addr 172.21.15.7:0/2538200880 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:34.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.528+0000 7fda72ffd640 10 monclient(hunting): get_auth_request con 0x7fda740a9d10 auth_method 0 2024-01-21T03:56:34.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.528+0000 7fda72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:34.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.528+0000 7fda72ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:34.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.528+0000 7fda72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:34.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.528+0000 7fda72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:34.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.528+0000 7fda72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:34.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.528+0000 7fda72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:34.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda727fc640 1 -- 172.21.15.7:0/2538200880 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257356914 0 0) 0x7fda740a98c0 con 0x7fda740b2310 2024-01-21T03:56:34.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda727fc640 10 monclient(hunting): _negotiate 2024-01-21T03:56:34.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda727fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:34.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda727fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:34.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda727fc640 10 monclient(hunting): my global_id is 23221 2024-01-21T03:56:34.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda727fc640 1 -- 172.21.15.7:0/2538200880 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda5c003520 con 0x7fda740b2310 2024-01-21T03:56:34.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda72ffd640 10 monclient(hunting): handle_auth_done global_id 15906 payload 274 2024-01-21T03:56:34.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda72ffd640 1 -- 172.21.15.7:0/2538200880 >> v1:172.21.15.177:6789/0 conn(0x7fda740b2310 legacy=0x7fda7413f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda72ffd640 1 -- 172.21.15.7:0/2538200880 >> v1:172.21.15.7:6789/0 conn(0x7fda740a8b20 legacy=0x7fda74145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda72ffd640 10 monclient: _finish_hunting 0 2024-01-21T03:56:34.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda72ffd640 1 monclient: found mon.noname-a 2024-01-21T03:56:34.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:34.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda72ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:56:34.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.530+0000 7fda72ffd640 1 -- 172.21.15.7:0/2538200880 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda741463d0 con 0x7fda740a9d10 2024-01-21T03:56:34.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.533+0000 7fda72ffd640 1 --2- 172.21.15.7:0/2538200880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda740a9d10 0x7fda7413f080 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7fda68009d90 tx=0x7fda6802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1179e745ee95ffd0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:34.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda727fc640 1 -- 172.21.15.7:0/2538200880 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6802cd00 con 0x7fda740a9d10 2024-01-21T03:56:34.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:34.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:56:34.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda727fc640 10 monclient: dump: 2024-01-21T03:56:34.540 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:34.540 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:34.540 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.540 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.540 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:34.540 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:34.540 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:34.540 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:34.540 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda727fc640 10 monclient: _finish_auth 0 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda727fc640 10 monclient: _check_auth_tickets 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda727fc640 1 -- 172.21.15.7:0/2538200880 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6802ce60 con 0x7fda740a9d10 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda7a234640 5 monclient: authenticate success, global_id 15906 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda7a234640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda727fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda7a234640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda727fc640 1 -- 172.21.15.7:0/2538200880 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda68034660 con 0x7fda740a9d10 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda7a234640 4 set_mon_vals no callback set 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.539+0000 7fda727fc640 10 monclient: dump: 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:34.541 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:34.542 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.542 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.542 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:34.542 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:34.542 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:34.542 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:34.542 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:34.542 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.540+0000 7fda7a234640 10 monclient: shutdown 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.540+0000 7fda7a234640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.540+0000 7fda7a234640 1 -- 172.21.15.7:0/2538200880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda740a9d10 msgr2=0x7fda7413f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.540+0000 7fda7a234640 1 --2- 172.21.15.7:0/2538200880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda740a9d10 0x7fda7413f080 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7fda68009d90 tx=0x7fda6802bf50 comp rx=0 tx=0).stop 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.540+0000 7fda7a234640 1 -- 172.21.15.7:0/2538200880 shutdown_connections 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.540+0000 7fda7a234640 1 --2- 172.21.15.7:0/2538200880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda740a9d10 0x7fda7413f080 unknown :-1 s=CLOSED pgs=3099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.540+0000 7fda7a234640 1 -- 172.21.15.7:0/2538200880 >> 172.21.15.7:0/2538200880 conn(0x7fda74066fb0 msgr2=0x7fda740b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.540+0000 7fda7a234640 1 -- 172.21.15.7:0/2538200880 shutdown_connections 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.540+0000 7fda7a234640 1 -- 172.21.15.7:0/2538200880 wait complete. 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.541+0000 7fda7a234640 10 monclient: build_initial_monmap 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.541+0000 7fda7a234640 10 monclient: monmap: 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.541+0000 7fda7a234640 1 librados: starting msgr at 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.541+0000 7fda7a234640 1 librados: starting objecter 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.541+0000 7fda7a234640 1 Processor -- start 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.541+0000 7fda7a234640 1 -- start start 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.541+0000 7fda7a234640 1 librados: setting wanted keys 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.541+0000 7fda7a234640 1 librados: calling monclient init 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.541+0000 7fda7a234640 10 monclient: init 2024-01-21T03:56:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda740a8b20 0x7fda74142900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 10 monclient(hunting): picked mon.noname-b con 0x7fda740a8b20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda740a9d10 0x7fda74140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 10 monclient(hunting): picked mon.noname-a con 0x7fda740a9d10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fda740b2310 0x7fda74138c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 10 monclient(hunting): picked mon.noname-c con 0x7fda740b2310 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda74141460 con 0x7fda740a9d10 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda74136e70 con 0x7fda740a8b20 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda74132840 con 0x7fda740b2310 2024-01-21T03:56:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.542+0000 7fda7a234640 10 monclient(hunting): authenticate will time out at 3540.858144s 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda737fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda740a8b20 0x7fda74142900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda72ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda740a9d10 0x7fda74140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda72ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda740a9d10 0x7fda74140f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44756/0 (socket says 172.21.15.7:44756) 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda72ffd640 1 -- 172.21.15.7:0/2613286227 learned_addr learned my addr 172.21.15.7:0/2613286227 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda72ffd640 10 monclient(hunting): get_auth_request con 0x7fda740a9d10 auth_method 0 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda72ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda737fe640 10 monclient(hunting): get_auth_request con 0x7fda740a8b20 auth_method 0 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda737fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda737fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.543+0000 7fda737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.544+0000 7fda73fff640 1 --2- 172.21.15.7:0/2613286227 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fda740b2310 0x7fda74138c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.545+0000 7fda73fff640 10 monclient(hunting): get_auth_request con 0x7fda740b2310 auth_method 0 2024-01-21T03:56:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.545+0000 7fda73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.545+0000 7fda73fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.545+0000 7fda73fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.545+0000 7fda73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.545+0000 7fda73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.545+0000 7fda73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:34.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.546+0000 7fda737fe640 10 monclient(hunting): handle_auth_done global_id 23224 payload 824 2024-01-21T03:56:34.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.546+0000 7fda737fe640 1 -- 172.21.15.7:0/2613286227 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fda740b2310 msgr2=0x7fda74138c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.546+0000 7fda737fe640 1 --2- 172.21.15.7:0/2613286227 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fda740b2310 0x7fda74138c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.546+0000 7fda737fe640 1 -- 172.21.15.7:0/2613286227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda740a9d10 msgr2=0x7fda74140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.546+0000 7fda737fe640 1 --2- 172.21.15.7:0/2613286227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda740a9d10 0x7fda74140f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.546+0000 7fda737fe640 10 monclient: _finish_hunting 0 2024-01-21T03:56:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.546+0000 7fda737fe640 1 monclient: found mon.noname-b 2024-01-21T03:56:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.546+0000 7fda737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.546+0000 7fda737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:56:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.546+0000 7fda737fe640 1 -- 172.21.15.7:0/2613286227 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda68009a70 con 0x7fda740a8b20 2024-01-21T03:56:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.548+0000 7fda737fe640 1 --2- 172.21.15.7:0/2613286227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda740a8b20 0x7fda74142900 secure :-1 s=READY pgs=5829 cs=0 l=1 rev1=1 crypto rx=0x7fda6400c370 tx=0x7fda6400c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.548+0000 7fda5bfff640 10 client.?.objecter ms_handle_connect 0x7fda740a8b20 2024-01-21T03:56:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.548+0000 7fda5bfff640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda5bfff640 1 -- 172.21.15.7:0/2613286227 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda640052d0 con 0x7fda740a8b20 2024-01-21T03:56:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda5bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda5bfff640 10 monclient: dump: 2024-01-21T03:56:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda5bfff640 10 monclient: _finish_auth 0 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda5bfff640 10 monclient: _check_auth_tickets 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda5bfff640 1 -- 172.21.15.7:0/2613286227 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda64005450 con 0x7fda740a8b20 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda5bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda5bfff640 1 -- 172.21.15.7:0/2613286227 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda64007260 con 0x7fda740a8b20 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda5bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda5bfff640 10 monclient: dump: 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda7a234640 5 monclient: authenticate success, global_id 23224 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda7a234640 10 monclient: _renew_subs 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda71ffb640 4 set_mon_vals no callback set 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda7a234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda7a234640 1 -- 172.21.15.7:0/2613286227 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda74136b90 con 0x7fda740a8b20 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda7a234640 10 client.23224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda7a234640 10 monclient: _renew_subs 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda7a234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda7a234640 1 -- 172.21.15.7:0/2613286227 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda74138820 con 0x7fda740a8b20 2024-01-21T03:56:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.552+0000 7fda7a234640 1 librados: init done 2024-01-21T03:56:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.553+0000 7fda7a234640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.553+0000 7fda7a234640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.553+0000 7fda7a234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.553+0000 7fda7a234640 1 -- 172.21.15.7:0/2613286227 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda741382c0 con 0x7fda740a8b20 2024-01-21T03:56:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.806+0000 7fda5bfff640 1 -- 172.21.15.7:0/2613286227 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fda640073c0 con 0x7fda740a8b20 2024-01-21T03:56:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.807+0000 7fda5bfff640 1 --2- 172.21.15.7:0/2613286227 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fda6006c8b0 0x7fda6006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.807+0000 7fda72ffd640 1 --2- 172.21.15.7:0/2613286227 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fda6006c8b0 0x7fda6006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.807+0000 7fda72ffd640 10 monclient: get_auth_request con 0x7fda6006c8b0 auth_method 0 2024-01-21T03:56:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.807+0000 7fda5bfff640 10 client.23224.objecter ms_handle_connect 0x7fda6006c8b0 2024-01-21T03:56:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.807+0000 7fda72ffd640 1 --2- 172.21.15.7:0/2613286227 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fda6006c8b0 0x7fda6006ed70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fda74132c00 tx=0x7fda68030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:34.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.809+0000 7fda5bfff640 1 -- 172.21.15.7:0/2613286227 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7fda64057a30 con 0x7fda740a8b20 2024-01-21T03:56:34.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.809+0000 7fda5bfff640 10 client.23224.objecter ms_dispatch 0x7fda74000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:34.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.809+0000 7fda5bfff640 3 client.23224.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:34.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.809+0000 7fda5bfff640 3 client.23224.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:34.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.810+0000 7fda5bfff640 20 client.23224.objecter dump_active .. 0 homeless 2024-01-21T03:56:34.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.819+0000 7fda5bfff640 1 -- 172.21.15.7:0/2613286227 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fda64053d70 con 0x7fda740a8b20 2024-01-21T03:56:34.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.819+0000 7fda5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:34.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:34.819+0000 7fda5bfff640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:35.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.026+0000 7fda7a234640 1 -- 172.21.15.7:0/2613286227 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fda740a9150 con 0x7fda6006c8b0 2024-01-21T03:56:35.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.027+0000 7fda5bfff640 1 -- 172.21.15.7:0/2613286227 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fda68042070 con 0x7fda6006c8b0 2024-01-21T03:56:35.027 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 6 to deep-scrub 2024-01-21T03:56:35.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.028+0000 7fda7a234640 10 librados: watch_flush enter 2024-01-21T03:56:35.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.028+0000 7fda7a234640 10 librados: watch_flush exit 2024-01-21T03:56:35.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.028+0000 7fda7a234640 20 client.23224.objecter shutdown clearing up homeless session... 2024-01-21T03:56:35.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.028+0000 7fda7a234640 10 client.23224.objecter successfully canceled tick 2024-01-21T03:56:35.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.028+0000 7fda7a234640 1 -- 172.21.15.7:0/2613286227 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fda6006c8b0 msgr2=0x7fda6006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.028+0000 7fda7a234640 1 --2- 172.21.15.7:0/2613286227 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fda6006c8b0 0x7fda6006ed70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fda74132c00 tx=0x7fda68030000 comp rx=0 tx=0).stop 2024-01-21T03:56:35.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.028+0000 7fda7a234640 10 monclient: shutdown 2024-01-21T03:56:35.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.028+0000 7fda7a234640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:35.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.028+0000 7fda7a234640 1 -- 172.21.15.7:0/2613286227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda740a8b20 msgr2=0x7fda74142900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.028+0000 7fda7a234640 1 --2- 172.21.15.7:0/2613286227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda740a8b20 0x7fda74142900 secure :-1 s=READY pgs=5829 cs=0 l=1 rev1=1 crypto rx=0x7fda6400c370 tx=0x7fda6400c830 comp rx=0 tx=0).stop 2024-01-21T03:56:35.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.029+0000 7fda7a234640 1 -- 172.21.15.7:0/2613286227 shutdown_connections 2024-01-21T03:56:35.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.029+0000 7fda7a234640 1 --2- 172.21.15.7:0/2613286227 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fda6006c8b0 0x7fda6006ed70 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.029+0000 7fda7a234640 1 --2- 172.21.15.7:0/2613286227 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fda740b2310 0x7fda74138c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.029+0000 7fda7a234640 1 --2- 172.21.15.7:0/2613286227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda740a9d10 0x7fda74140f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.029+0000 7fda7a234640 1 --2- 172.21.15.7:0/2613286227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda740a8b20 0x7fda74142900 unknown :-1 s=CLOSED pgs=5829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.029+0000 7fda7a234640 1 -- 172.21.15.7:0/2613286227 >> 172.21.15.7:0/2613286227 conn(0x7fda74066fb0 msgr2=0x7fda740b3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:35.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.029+0000 7fda7a234640 1 -- 172.21.15.7:0/2613286227 shutdown_connections 2024-01-21T03:56:35.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.029+0000 7fda7a234640 1 -- 172.21.15.7:0/2613286227 wait complete. 2024-01-21T03:56:35.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.029+0000 7fda7a234640 1 librados: shutdown 2024-01-21T03:56:35.040 INFO:tasks.ceph:Scrubbing osd.7 2024-01-21T03:56:35.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-01-21T03:56:35.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.144+0000 7fac6eb6c640 10 monclient: get_monmap_and_config 2024-01-21T03:56:35.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.144+0000 7fac6eb6c640 10 monclient: build_initial_monmap 2024-01-21T03:56:35.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.144+0000 7fac6eb6c640 10 monclient: monmap: 2024-01-21T03:56:35.144 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:35.145075+0000 2024-01-21T03:56:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:35.145075+0000 2024-01-21T03:56:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:35.145 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.144+0000 7fac6eb6c640 1 Processor -- start 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.144+0000 7fac6eb6c640 1 -- start start 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.144+0000 7fac6eb6c640 10 monclient: init 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac680a8a40 0x7fac68145b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 10 monclient(hunting): picked mon.noname-a con 0x7fac680a8a40 addr v2:172.21.15.7:3300/0 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fac680b2210 0x7fac6813f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 10 monclient(hunting): picked mon.noname-c con 0x7fac680b2210 addr v2:172.21.15.177:3300/0 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fac680a8e20 0x7fac6813f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 10 monclient(hunting): picked mon.noname-b con 0x7fac680a8e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac680aa7b0 con 0x7fac680a8a40 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6813fc40 con 0x7fac680b2210 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac68145700 con 0x7fac680a8e20 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6eb6c640 10 monclient(hunting): authenticate will time out at 3541.461195s 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.145+0000 7fac6c908640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac680a8a40 0x7fac68145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6d109640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fac680a8e20 0x7fac6813f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac67fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fac680b2210 0x7fac6813f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6d109640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fac680a8e20 0x7fac6813f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37230/0 (socket says 172.21.15.7:37230) 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6d109640 1 -- 172.21.15.7:0/1256092528 learned_addr learned my addr 172.21.15.7:0/1256092528 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6d109640 10 monclient(hunting): get_auth_request con 0x7fac680a8e20 auth_method 0 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6d109640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6d109640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6d109640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6c908640 10 monclient(hunting): get_auth_request con 0x7fac680a8a40 auth_method 0 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6c908640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6c908640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6c908640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac67fff640 10 monclient(hunting): get_auth_request con 0x7fac680b2210 auth_method 0 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac67fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac67fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6d109640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6d109640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6d109640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6c908640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6c908640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.146+0000 7fac6c908640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.147+0000 7fac67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.147+0000 7fac67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.147+0000 7fac67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.148+0000 7fac6c908640 10 monclient(hunting): handle_auth_done global_id 15915 payload 274 2024-01-21T03:56:35.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.148+0000 7fac6c908640 1 -- 172.21.15.7:0/1256092528 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fac680a8e20 msgr2=0x7fac6813f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:35.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.148+0000 7fac6c908640 1 --2- 172.21.15.7:0/1256092528 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fac680a8e20 0x7fac6813f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.148+0000 7fac6c908640 1 -- 172.21.15.7:0/1256092528 >> v2:172.21.15.177:3300/0 conn(0x7fac680b2210 msgr2=0x7fac6813f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:35.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.148+0000 7fac6c908640 1 --2- 172.21.15.7:0/1256092528 >> v2:172.21.15.177:3300/0 conn(0x7fac680b2210 0x7fac6813f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.148+0000 7fac6c908640 10 monclient: _finish_hunting 0 2024-01-21T03:56:35.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.148+0000 7fac6c908640 1 monclient: found mon.noname-a 2024-01-21T03:56:35.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.148+0000 7fac6c908640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:35.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.148+0000 7fac6c908640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-01-21T03:56:35.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.148+0000 7fac6c908640 1 -- 172.21.15.7:0/1256092528 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac680aa9b0 con 0x7fac680a8a40 2024-01-21T03:56:35.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.148+0000 7fac67fff640 1 --2- 172.21.15.7:0/1256092528 >> v2:172.21.15.177:3300/0 conn(0x7fac680b2210 0x7fac6813f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T03:56:35.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.150+0000 7fac6c908640 1 --2- 172.21.15.7:0/1256092528 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac680a8a40 0x7fac68145b90 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7fac680b1bb0 tx=0x7fac5802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c086fd86c0e73 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:35.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.152+0000 7fac677fe640 1 -- 172.21.15.7:0/1256092528 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5802e070 con 0x7fac680a8a40 2024-01-21T03:56:35.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.152+0000 7fac677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:35.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.152+0000 7fac677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T03:56:35.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.152+0000 7fac677fe640 10 monclient: dump: 2024-01-21T03:56:35.153 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:35.153 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:35.153 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.153 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.154 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:35.154 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.154 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:35.154 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:35.154 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:35.154 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.152+0000 7fac677fe640 10 monclient: _finish_auth 0 2024-01-21T03:56:35.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.152+0000 7fac677fe640 10 monclient: _check_auth_tickets 2024-01-21T03:56:35.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.152+0000 7fac677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:35.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.152+0000 7fac6eb6c640 5 monclient: authenticate success, global_id 15915 2024-01-21T03:56:35.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.152+0000 7fac6eb6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T03:56:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.154+0000 7fac677fe640 1 -- 172.21.15.7:0/1256092528 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac580315a0 con 0x7fac680a8a40 2024-01-21T03:56:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.154+0000 7fac677fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.154+0000 7fac677fe640 1 -- 172.21.15.7:0/1256092528 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58031930 con 0x7fac680a8a40 2024-01-21T03:56:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.154+0000 7fac677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.154+0000 7fac677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T03:56:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.154+0000 7fac677fe640 10 monclient: dump: 2024-01-21T03:56:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.154+0000 7fac6eb6c640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.154+0000 7fac6eb6c640 4 set_mon_vals no callback set 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.155+0000 7fac6eb6c640 10 monclient: shutdown 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.155+0000 7fac6eb6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.155+0000 7fac6eb6c640 1 -- 172.21.15.7:0/1256092528 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac680a8a40 msgr2=0x7fac68145b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.155+0000 7fac6eb6c640 1 --2- 172.21.15.7:0/1256092528 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac680a8a40 0x7fac68145b90 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7fac680b1bb0 tx=0x7fac5802b840 comp rx=0 tx=0).stop 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.155+0000 7fac6eb6c640 1 -- 172.21.15.7:0/1256092528 shutdown_connections 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.155+0000 7fac6eb6c640 1 --2- 172.21.15.7:0/1256092528 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fac680a8e20 0x7fac6813f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.155+0000 7fac6eb6c640 1 --2- 172.21.15.7:0/1256092528 >> v2:172.21.15.177:3300/0 conn(0x7fac680b2210 0x7fac6813f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.155+0000 7fac6eb6c640 1 --2- 172.21.15.7:0/1256092528 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac680a8a40 0x7fac68145b90 unknown :-1 s=CLOSED pgs=3100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.155+0000 7fac6eb6c640 1 -- 172.21.15.7:0/1256092528 >> 172.21.15.7:0/1256092528 conn(0x7fac68066f10 msgr2=0x7fac680b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.155+0000 7fac6eb6c640 1 -- 172.21.15.7:0/1256092528 shutdown_connections 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.155+0000 7fac6eb6c640 1 -- 172.21.15.7:0/1256092528 wait complete. 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.155+0000 7fac6eb6c640 10 monclient: build_initial_monmap 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.155+0000 7fac6eb6c640 10 monclient: monmap: 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:35.158 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.156+0000 7fac6eb6c640 1 librados: starting msgr at 2024-01-21T03:56:35.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.156+0000 7fac6eb6c640 1 librados: starting objecter 2024-01-21T03:56:35.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.156+0000 7fac6eb6c640 1 Processor -- start 2024-01-21T03:56:35.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.156+0000 7fac6eb6c640 1 -- start start 2024-01-21T03:56:35.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.156+0000 7fac6eb6c640 1 librados: setting wanted keys 2024-01-21T03:56:35.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.156+0000 7fac6eb6c640 1 librados: calling monclient init 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.156+0000 7fac6eb6c640 10 monclient: init 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fac680a8a40 0x7fac68143b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 10 monclient(hunting): picked mon.noname-c con 0x7fac680a8a40 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac680a8e20 0x7fac68144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 10 monclient(hunting): picked mon.noname-b con 0x7fac680a8e20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac680b2210 0x7fac681d76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 10 monclient(hunting): picked mon.noname-a con 0x7fac680b2210 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac680abec0 con 0x7fac680b2210 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac681d7180 con 0x7fac680a8e20 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac68144c40 con 0x7fac680a8a40 2024-01-21T03:56:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6eb6c640 10 monclient(hunting): authenticate will time out at 3541.472928s 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6d109640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac680b2210 0x7fac681d76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.157+0000 7fac6c908640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fac680a8a40 0x7fac68143b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6d109640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac680b2210 0x7fac681d76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44770/0 (socket says 172.21.15.7:44770) 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6d109640 1 -- 172.21.15.7:0/2171306419 learned_addr learned my addr 172.21.15.7:0/2171306419 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6d109640 10 monclient(hunting): get_auth_request con 0x7fac680b2210 auth_method 0 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6d109640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6d109640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6d109640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6c908640 10 monclient(hunting): get_auth_request con 0x7fac680a8a40 auth_method 0 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6c908640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6c908640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6c908640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6d109640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6d109640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6d109640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6c908640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6c908640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.158+0000 7fac6c908640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.159+0000 7fac67fff640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac680a8e20 0x7fac68144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.160+0000 7fac67fff640 10 monclient(hunting): get_auth_request con 0x7fac680a8e20 auth_method 0 2024-01-21T03:56:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.160+0000 7fac67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.160+0000 7fac67fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.160+0000 7fac67fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.160+0000 7fac67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.160+0000 7fac67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.160+0000 7fac67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.163+0000 7fac6c908640 10 monclient(hunting): handle_auth_done global_id 18071 payload 824 2024-01-21T03:56:35.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.163+0000 7fac6c908640 1 -- 172.21.15.7:0/2171306419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac680a8e20 msgr2=0x7fac68144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.163+0000 7fac6c908640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac680a8e20 0x7fac68144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.163+0000 7fac6c908640 1 -- 172.21.15.7:0/2171306419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac680b2210 msgr2=0x7fac681d76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.163+0000 7fac6c908640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac680b2210 0x7fac681d76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.163+0000 7fac6c908640 10 monclient: _finish_hunting 0 2024-01-21T03:56:35.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.163+0000 7fac6c908640 1 monclient: found mon.noname-c 2024-01-21T03:56:35.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.163+0000 7fac6c908640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:35.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.163+0000 7fac6c908640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:56:35.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.163+0000 7fac6c908640 1 -- 172.21.15.7:0/2171306419 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c009400 con 0x7fac680a8a40 2024-01-21T03:56:35.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.163+0000 7fac6d109640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac680b2210 0x7fac681d76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T03:56:35.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.166+0000 7fac6c908640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fac680a8a40 0x7fac68143b50 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7fac58033000 tx=0x7fac580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:35.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.166+0000 7fac657fa640 10 client.?.objecter ms_handle_connect 0x7fac680a8a40 2024-01-21T03:56:35.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.166+0000 7fac657fa640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:35.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac657fa640 1 -- 172.21.15.7:0/2171306419 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58032630 con 0x7fac680a8a40 2024-01-21T03:56:35.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:35.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac657fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:56:35.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac657fa640 10 monclient: dump: 2024-01-21T03:56:35.172 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:35.172 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:35.172 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac657fa640 10 monclient: _finish_auth 0 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac657fa640 10 monclient: _check_auth_tickets 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac657fa640 1 -- 172.21.15.7:0/2171306419 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac580327b0 con 0x7fac680a8a40 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac657fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac657fa640 1 -- 172.21.15.7:0/2171306419 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58032b60 con 0x7fac680a8a40 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac6eb6c640 5 monclient: authenticate success, global_id 18071 2024-01-21T03:56:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac6eb6c640 10 monclient: _renew_subs 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac6eb6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac6eb6c640 1 -- 172.21.15.7:0/2171306419 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac58009400 con 0x7fac680a8a40 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac66ffd640 4 set_mon_vals no callback set 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac657fa640 10 monclient: dump: 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:35.174 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:35.175 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac6eb6c640 10 client.18071.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac6eb6c640 10 monclient: _renew_subs 2024-01-21T03:56:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac6eb6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac6eb6c640 1 -- 172.21.15.7:0/2171306419 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac681d7dc0 con 0x7fac680a8a40 2024-01-21T03:56:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.171+0000 7fac6eb6c640 1 librados: init done 2024-01-21T03:56:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.172+0000 7fac6eb6c640 10 client.18071.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.172+0000 7fac6eb6c640 15 client.18071.objecter _session_command_op_assign -1 1 2024-01-21T03:56:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.172+0000 7fac6eb6c640 20 client.18071.objecter _get_session osd=-1 returning homeless 2024-01-21T03:56:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.172+0000 7fac6eb6c640 10 client.18071.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.172+0000 7fac6eb6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:35.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.172+0000 7fac6eb6c640 1 -- 172.21.15.7:0/2171306419 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fac680aacf0 con 0x7fac680a8a40 2024-01-21T03:56:35.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.191+0000 7fac657fa640 1 -- 172.21.15.7:0/2171306419 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fac58032cc0 con 0x7fac680a8a40 2024-01-21T03:56:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.192+0000 7fac657fa640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fac5406c8b0 0x7fac5406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.192+0000 7fac67fff640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fac5406c8b0 0x7fac5406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.192+0000 7fac67fff640 10 monclient: get_auth_request con 0x7fac5406c8b0 auth_method 0 2024-01-21T03:56:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.192+0000 7fac67fff640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fac5406c8b0 0x7fac5406ed70 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fac6813bc60 tx=0x7fac5c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.192+0000 7fac657fa640 10 client.18071.objecter ms_handle_connect 0x7fac5406c8b0 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 1 -- 172.21.15.7:0/2171306419 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7fac5807aad0 con 0x7fac680a8a40 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 10 client.18071.objecter ms_dispatch 0x7fac68000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 3 client.18071.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 3 client.18071.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 10 client.18071.objecter checking command 1 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x7fac54074610 0x7fac54076ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 20 client.18071.objecter _get_session s=0x7fac54073f40 osd=7 3 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 20 client.18071.objecter put_session s=0x7fac54073f40 osd=7 3 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 15 client.18071.objecter _session_command_op_remove -1 1 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 20 client.18071.objecter _get_session s=0x7fac54073f40 osd=7 3 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 20 client.18071.objecter get_session s=0x7fac54073f40 osd=7 3 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 15 client.18071.objecter _session_command_op_assign 7 1 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 20 client.18071.objecter put_session s=0x7fac54073f40 osd=7 4 2024-01-21T03:56:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 10 client.18071.objecter _send_command 1 2024-01-21T03:56:35.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 1 -- 172.21.15.7:0/2171306419 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac54077180 con 0x7fac54074610 2024-01-21T03:56:35.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 20 client.18071.objecter dump_active .. 0 homeless 2024-01-21T03:56:35.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 1 -- 172.21.15.7:0/2171306419 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=1294) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac58076710 con 0x7fac680a8a40 2024-01-21T03:56:35.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.195+0000 7fac657fa640 10 monclient: handle_get_version_reply finishing 1 version 1294 2024-01-21T03:56:35.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.196+0000 7fac6d109640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x7fac54074610 0x7fac54076ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.197+0000 7fac6d109640 10 monclient: get_auth_request con 0x7fac54074610 auth_method 0 2024-01-21T03:56:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.199+0000 7fac657fa640 10 client.18071.objecter ms_handle_connect 0x7fac54074610 2024-01-21T03:56:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.199+0000 7fac6d109640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x7fac54074610 0x7fac54076ab0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:35.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.225+0000 7fac657fa640 1 -- 172.21.15.7:0/2171306419 <== osd.7 v2:172.21.15.177:6823/4069998673 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fac54077180 con 0x7fac54074610 2024-01-21T03:56:35.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.225+0000 7fac657fa640 10 client.18071.objecter ms_dispatch 0x7fac68000b90 command_reply(tid 1: 0 ) v1 2024-01-21T03:56:35.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.225+0000 7fac657fa640 10 client.18071.objecter _finish_command 1 = system:0 2024-01-21T03:56:35.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.225+0000 7fac657fa640 20 client.18071.objecter put_session s=0x7fac54073f40 osd=7 4 2024-01-21T03:56:35.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.225+0000 7fac657fa640 15 client.18071.objecter _session_command_op_remove 7 1 2024-01-21T03:56:35.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.238+0000 7fac6eb6c640 10 client.18071.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T03:56:35.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.238+0000 7fac6eb6c640 15 client.18071.objecter _session_command_op_assign -1 2 2024-01-21T03:56:35.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.238+0000 7fac6eb6c640 20 client.18071.objecter _get_session s=0x7fac54073f40 osd=7 3 2024-01-21T03:56:35.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.239+0000 7fac6eb6c640 20 client.18071.objecter put_session s=0x7fac54073f40 osd=7 3 2024-01-21T03:56:35.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.239+0000 7fac6eb6c640 20 client.18071.objecter _get_session s=0x7fac54073f40 osd=7 3 2024-01-21T03:56:35.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.239+0000 7fac6eb6c640 15 client.18071.objecter _session_command_op_remove -1 2 2024-01-21T03:56:35.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.239+0000 7fac6eb6c640 20 client.18071.objecter get_session s=0x7fac54073f40 osd=7 3 2024-01-21T03:56:35.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.239+0000 7fac6eb6c640 15 client.18071.objecter _session_command_op_assign 7 2 2024-01-21T03:56:35.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.239+0000 7fac6eb6c640 20 client.18071.objecter put_session s=0x7fac54073f40 osd=7 4 2024-01-21T03:56:35.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.239+0000 7fac6eb6c640 10 client.18071.objecter _send_command 2 2024-01-21T03:56:35.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.239+0000 7fac6eb6c640 1 -- 172.21.15.7:0/2171306419 --> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fac680aa1c0 con 0x7fac54074610 2024-01-21T03:56:35.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.253+0000 7fac657fa640 1 -- 172.21.15.7:0/2171306419 <== osd.7 v2:172.21.15.177:6823/4069998673 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fac680aa1c0 con 0x7fac54074610 2024-01-21T03:56:35.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.253+0000 7fac657fa640 10 client.18071.objecter ms_dispatch 0x7fac68000b90 command_reply(tid 2: 0 ) v1 2024-01-21T03:56:35.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.253+0000 7fac657fa640 10 client.18071.objecter _finish_command 2 = system:0 2024-01-21T03:56:35.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.253+0000 7fac657fa640 20 client.18071.objecter put_session s=0x7fac54073f40 osd=7 4 2024-01-21T03:56:35.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.253+0000 7fac657fa640 15 client.18071.objecter _session_command_op_remove 7 2 2024-01-21T03:56:35.254 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-01-21T03:56:35.254 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T03:56:35.255 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-01-21T03:56:35.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 10 librados: watch_flush enter 2024-01-21T03:56:35.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 10 librados: watch_flush exit 2024-01-21T03:56:35.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 10 client.18071.objecter close_session for osd.7 2024-01-21T03:56:35.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 1 -- 172.21.15.7:0/2171306419 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x7fac54074610 msgr2=0x7fac54076ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.177:6823/4069998673,v1:172.21.15.177:6825/4069998673] conn(0x7fac54074610 0x7fac54076ab0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 20 client.18071.objecter put_session s=0x7fac54073f40 osd=7 1 2024-01-21T03:56:35.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 20 client.18071.objecter shutdown clearing up homeless session... 2024-01-21T03:56:35.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 10 client.18071.objecter successfully canceled tick 2024-01-21T03:56:35.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 1 -- 172.21.15.7:0/2171306419 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fac5406c8b0 msgr2=0x7fac5406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fac5406c8b0 0x7fac5406ed70 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fac6813bc60 tx=0x7fac5c009190 comp rx=0 tx=0).stop 2024-01-21T03:56:35.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 10 monclient: shutdown 2024-01-21T03:56:35.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:35.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 1 -- 172.21.15.7:0/2171306419 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fac680a8a40 msgr2=0x7fac68143b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 1 --2- 172.21.15.7:0/2171306419 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fac680a8a40 0x7fac68143b50 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7fac58033000 tx=0x7fac580046c0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6d109640 1 -- 172.21.15.7:0/2171306419 reap_dead start 2024-01-21T03:56:35.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 1 -- 172.21.15.7:0/2171306419 shutdown_connections 2024-01-21T03:56:35.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 1 -- 172.21.15.7:0/2171306419 >> 172.21.15.7:0/2171306419 conn(0x7fac68066f10 msgr2=0x7fac680b2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:35.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 1 -- 172.21.15.7:0/2171306419 shutdown_connections 2024-01-21T03:56:35.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 1 -- 172.21.15.7:0/2171306419 wait complete. 2024-01-21T03:56:35.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.254+0000 7fac6eb6c640 1 librados: shutdown 2024-01-21T03:56:35.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-01-21T03:56:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.368+0000 7fa8fb076640 10 monclient: get_monmap_and_config 2024-01-21T03:56:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.368+0000 7fa8fb076640 10 monclient: build_initial_monmap 2024-01-21T03:56:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.368+0000 7fa8fb076640 10 monclient: monmap: 2024-01-21T03:56:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:35.369666+0000 2024-01-21T03:56:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:35.369666+0000 2024-01-21T03:56:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:35.370 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.369+0000 7fa8fb076640 1 Processor -- start 2024-01-21T03:56:35.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.369+0000 7fa8fb076640 1 -- start start 2024-01-21T03:56:35.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.369+0000 7fa8fb076640 10 monclient: init 2024-01-21T03:56:35.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.369+0000 7fa8fb076640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.369+0000 7fa8fb076640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.369+0000 7fa8fb076640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa8f40a8ab0 0x7fa8f4142640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8fb076640 10 monclient(hunting): picked mon.noname-b con 0x7fa8f40a8ab0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8fb076640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8f40b2310 addr v1:172.21.15.7:6789/0 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8fb076640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa8f40a8e90 0x7fa8f4142b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8fb076640 10 monclient(hunting): picked mon.noname-c con 0x7fa8f40a8e90 addr v2:172.21.15.177:3300/0 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8fb076640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f40aa7d0 con 0x7fa8f40b2310 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8fb076640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8fb076640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f40aaa20 con 0x7fa8f40a8e90 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8fb076640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8fb076640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f40aad00 con 0x7fa8f40a8ab0 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8fb076640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8fb076640 10 monclient(hunting): authenticate will time out at 3541.685808s 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8f8e12640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa8f40a8ab0 0x7fa8f4142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8f3fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa8f40b2310 0x7fa8f413f130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56436/0 (socket says 172.21.15.7:56436) 2024-01-21T03:56:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8f3fff640 1 -- 172.21.15.7:0/462213498 learned_addr learned my addr 172.21.15.7:0/462213498 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8f9613640 1 --2- 172.21.15.7:0/462213498 >> v2:172.21.15.177:3300/0 conn(0x7fa8f40a8e90 0x7fa8f4142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8f8e12640 10 monclient(hunting): get_auth_request con 0x7fa8f40a8ab0 auth_method 0 2024-01-21T03:56:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8f8e12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8f8e12640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.370+0000 7fa8f8e12640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.371+0000 7fa8f9613640 10 monclient(hunting): get_auth_request con 0x7fa8f40a8e90 auth_method 0 2024-01-21T03:56:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.371+0000 7fa8f9613640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.371+0000 7fa8f9613640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.371+0000 7fa8f9613640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.371+0000 7fa8f8e12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.371+0000 7fa8f8e12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.371+0000 7fa8f8e12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.371+0000 7fa8f9613640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.371+0000 7fa8f9613640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.371+0000 7fa8f9613640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f37fe640 1 -- 172.21.15.7:0/462213498 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020784597 0 0) 0x7fa8f40aa7d0 con 0x7fa8f40b2310 2024-01-21T03:56:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f37fe640 10 monclient(hunting): _negotiate 2024-01-21T03:56:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f37fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f37fe640 10 monclient(hunting): my global_id is 15921 2024-01-21T03:56:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f37fe640 1 -- 172.21.15.7:0/462213498 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8dc003520 con 0x7fa8f40b2310 2024-01-21T03:56:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f9613640 10 monclient(hunting): handle_auth_done global_id 23233 payload 274 2024-01-21T03:56:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f9613640 1 -- 172.21.15.7:0/462213498 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa8f40a8ab0 msgr2=0x7fa8f4142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f9613640 1 --2- 172.21.15.7:0/462213498 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa8f40a8ab0 0x7fa8f4142640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f9613640 1 -- 172.21.15.7:0/462213498 >> v1:172.21.15.7:6789/0 conn(0x7fa8f40b2310 legacy=0x7fa8f413f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f9613640 10 monclient: _finish_hunting 0 2024-01-21T03:56:35.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f9613640 1 monclient: found mon.noname-c 2024-01-21T03:56:35.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f9613640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:35.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f9613640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:56:35.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.373+0000 7fa8f9613640 1 -- 172.21.15.7:0/462213498 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f40aafe0 con 0x7fa8f40a8e90 2024-01-21T03:56:35.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.375+0000 7fa8f9613640 1 --2- 172.21.15.7:0/462213498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8f40a8e90 0x7fa8f4142b80 secure :-1 s=READY pgs=5830 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009dc0 tx=0x7fa8ec02be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ff33f4415b16e43 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8f37fe640 1 -- 172.21.15.7:0/462213498 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec02cc40 con 0x7fa8f40a8e90 2024-01-21T03:56:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:56:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8f37fe640 10 monclient: dump: 2024-01-21T03:56:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8f37fe640 10 monclient: _finish_auth 0 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8f37fe640 10 monclient: _check_auth_tickets 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8f37fe640 1 -- 172.21.15.7:0/462213498 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8ec02cda0 con 0x7fa8f40a8e90 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8f37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8f37fe640 1 -- 172.21.15.7:0/462213498 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec034660 con 0x7fa8f40a8e90 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8fb076640 5 monclient: authenticate success, global_id 23233 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8fb076640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8fb076640 4 set_mon_vals no callback set 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.379+0000 7fa8f37fe640 10 monclient: dump: 2024-01-21T03:56:35.382 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:35.382 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:35.382 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.382 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.382 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:35.382 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.382 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:35.382 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:35.382 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:35.382 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.380+0000 7fa8fb076640 10 monclient: shutdown 2024-01-21T03:56:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.380+0000 7fa8fb076640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.380+0000 7fa8fb076640 1 -- 172.21.15.7:0/462213498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8f40a8e90 msgr2=0x7fa8f4142b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.380+0000 7fa8fb076640 1 --2- 172.21.15.7:0/462213498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8f40a8e90 0x7fa8f4142b80 secure :-1 s=READY pgs=5830 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009dc0 tx=0x7fa8ec02be90 comp rx=0 tx=0).stop 2024-01-21T03:56:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.380+0000 7fa8fb076640 1 -- 172.21.15.7:0/462213498 shutdown_connections 2024-01-21T03:56:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.380+0000 7fa8fb076640 1 --2- 172.21.15.7:0/462213498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8f40a8e90 0x7fa8f4142b80 unknown :-1 s=CLOSED pgs=5830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.380+0000 7fa8fb076640 1 --2- 172.21.15.7:0/462213498 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa8f40a8ab0 0x7fa8f4142640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.380+0000 7fa8fb076640 1 -- 172.21.15.7:0/462213498 >> 172.21.15.7:0/462213498 conn(0x7fa8f4066f60 msgr2=0x7fa8f40b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.380+0000 7fa8fb076640 1 -- 172.21.15.7:0/462213498 shutdown_connections 2024-01-21T03:56:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.380+0000 7fa8fb076640 1 -- 172.21.15.7:0/462213498 wait complete. 2024-01-21T03:56:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.381+0000 7fa8fb076640 10 monclient: build_initial_monmap 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.381+0000 7fa8fb076640 10 monclient: monmap: 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.381+0000 7fa8fb076640 1 librados: starting msgr at 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.381+0000 7fa8fb076640 1 librados: starting objecter 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.381+0000 7fa8fb076640 1 Processor -- start 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.381+0000 7fa8fb076640 1 -- start start 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.381+0000 7fa8fb076640 1 librados: setting wanted keys 2024-01-21T03:56:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.381+0000 7fa8fb076640 1 librados: calling monclient init 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.381+0000 7fa8fb076640 10 monclient: init 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa8f40a8ab0 0x7fa8f4135470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 10 monclient(hunting): picked mon.noname-c con 0x7fa8f40a8ab0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8f40a8e90 0x7fa8f4132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 10 monclient(hunting): picked mon.noname-a con 0x7fa8f40a8e90 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8f40b2310 0x7fa8f4132eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 10 monclient(hunting): picked mon.noname-b con 0x7fa8f40b2310 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4135f90 con 0x7fa8f40a8e90 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4137640 con 0x7fa8f40b2310 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f41378b0 con 0x7fa8f40a8ab0 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8fb076640 10 monclient(hunting): authenticate will time out at 3541.698123s 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8f8e12640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa8f40a8ab0 0x7fa8f4135470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.382+0000 7fa8f3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8f40a8e90 0x7fa8f4132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.383+0000 7fa8f8e12640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa8f40a8ab0 0x7fa8f4135470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:37252/0 (socket says 172.21.15.7:37252) 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.383+0000 7fa8f8e12640 1 -- 172.21.15.7:0/646729171 learned_addr learned my addr 172.21.15.7:0/646729171 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.383+0000 7fa8f8e12640 10 monclient(hunting): get_auth_request con 0x7fa8f40a8ab0 auth_method 0 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.383+0000 7fa8f8e12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.383+0000 7fa8f8e12640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.383+0000 7fa8f8e12640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.383+0000 7fa8f3fff640 10 monclient(hunting): get_auth_request con 0x7fa8f40a8e90 auth_method 0 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.383+0000 7fa8f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.383+0000 7fa8f3fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.383+0000 7fa8f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.383+0000 7fa8f8e12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.383+0000 7fa8f8e12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.383+0000 7fa8f8e12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.384+0000 7fa8f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.384+0000 7fa8f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.384+0000 7fa8f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.384+0000 7fa8f9613640 1 --2- 172.21.15.7:0/646729171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8f40b2310 0x7fa8f4132eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.385+0000 7fa8f9613640 10 monclient(hunting): get_auth_request con 0x7fa8f40b2310 auth_method 0 2024-01-21T03:56:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.385+0000 7fa8f9613640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.385+0000 7fa8f9613640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.385+0000 7fa8f9613640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.385+0000 7fa8f9613640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.385+0000 7fa8f9613640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.385+0000 7fa8f9613640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.386+0000 7fa8f8e12640 10 monclient(hunting): handle_auth_done global_id 18077 payload 824 2024-01-21T03:56:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.386+0000 7fa8f8e12640 1 -- 172.21.15.7:0/646729171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8f40b2310 msgr2=0x7fa8f4132eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.386+0000 7fa8f8e12640 1 --2- 172.21.15.7:0/646729171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8f40b2310 0x7fa8f4132eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.386+0000 7fa8f8e12640 1 -- 172.21.15.7:0/646729171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8f40a8e90 msgr2=0x7fa8f4132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.386+0000 7fa8f8e12640 1 --2- 172.21.15.7:0/646729171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8f40a8e90 0x7fa8f4132970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.386+0000 7fa8f8e12640 10 monclient: _finish_hunting 0 2024-01-21T03:56:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.386+0000 7fa8f8e12640 1 monclient: found mon.noname-c 2024-01-21T03:56:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.386+0000 7fa8f8e12640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.386+0000 7fa8f8e12640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-01-21T03:56:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.386+0000 7fa8f8e12640 1 -- 172.21.15.7:0/646729171 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec009a70 con 0x7fa8f40a8ab0 2024-01-21T03:56:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.388+0000 7fa8f8e12640 1 --2- 172.21.15.7:0/646729171 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa8f40a8ab0 0x7fa8f4135470 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400e9b0 tx=0x7fa8e400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.388+0000 7fa8f17fa640 10 client.?.objecter ms_handle_connect 0x7fa8f40a8ab0 2024-01-21T03:56:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.388+0000 7fa8f17fa640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:35.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f17fa640 1 -- 172.21.15.7:0/646729171 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4009670 con 0x7fa8f40a8ab0 2024-01-21T03:56:35.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:35.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:56:35.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f17fa640 10 monclient: dump: 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f17fa640 10 monclient: _finish_auth 0 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f17fa640 10 monclient: _check_auth_tickets 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f17fa640 1 -- 172.21.15.7:0/646729171 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e400c9a0 con 0x7fa8f40a8ab0 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f17fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f17fa640 1 -- 172.21.15.7:0/646729171 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e400cd50 con 0x7fa8f40a8ab0 2024-01-21T03:56:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f17fa640 10 monclient: dump: 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8fb076640 5 monclient: authenticate success, global_id 18077 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8fb076640 10 monclient: _renew_subs 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8fb076640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8fb076640 1 -- 172.21.15.7:0/646729171 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f4137b90 con 0x7fa8f40a8ab0 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8f2ffd640 4 set_mon_vals no callback set 2024-01-21T03:56:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8fb076640 10 client.18077.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:35.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8fb076640 10 monclient: _renew_subs 2024-01-21T03:56:35.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8fb076640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:35.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8fb076640 1 -- 172.21.15.7:0/646729171 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f4137fb0 con 0x7fa8f40a8ab0 2024-01-21T03:56:35.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.392+0000 7fa8fb076640 1 librados: init done 2024-01-21T03:56:35.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.393+0000 7fa8fb076640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:35.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.393+0000 7fa8fb076640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:35.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.393+0000 7fa8fb076640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-01-21T03:56:35.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.393+0000 7fa8fb076640 1 -- 172.21.15.7:0/646729171 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f404e970 con 0x7fa8f40a8ab0 2024-01-21T03:56:35.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.660+0000 7fa8f17fa640 1 -- 172.21.15.7:0/646729171 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa8e4013470 con 0x7fa8f40a8ab0 2024-01-21T03:56:35.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.660+0000 7fa8f17fa640 1 --2- 172.21.15.7:0/646729171 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fa8e006c890 0x7fa8e006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.661+0000 7fa8f3fff640 1 --2- 172.21.15.7:0/646729171 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fa8e006c890 0x7fa8e006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.661+0000 7fa8f3fff640 10 monclient: get_auth_request con 0x7fa8e006c890 auth_method 0 2024-01-21T03:56:35.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.661+0000 7fa8f3fff640 1 --2- 172.21.15.7:0/646729171 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fa8e006c890 0x7fa8e006ed50 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4137220 tx=0x7fa8e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:35.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.661+0000 7fa8f17fa640 10 client.18077.objecter ms_handle_connect 0x7fa8e006c890 2024-01-21T03:56:35.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.674+0000 7fa8f17fa640 1 -- 172.21.15.7:0/646729171 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7fa8e4058be0 con 0x7fa8f40a8ab0 2024-01-21T03:56:35.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.674+0000 7fa8f17fa640 10 client.18077.objecter ms_dispatch 0x7fa8f4000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:35.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.674+0000 7fa8f17fa640 3 client.18077.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:35.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.674+0000 7fa8f17fa640 3 client.18077.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:35.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.674+0000 7fa8f17fa640 20 client.18077.objecter dump_active .. 0 homeless 2024-01-21T03:56:35.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.674+0000 7fa8f17fa640 1 -- 172.21.15.7:0/646729171 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa8e4054f20 con 0x7fa8f40a8ab0 2024-01-21T03:56:35.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.674+0000 7fa8f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:35.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.674+0000 7fa8f17fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:35.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.858+0000 7fa8fb076640 1 -- 172.21.15.7:0/646729171 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8f4072860 con 0x7fa8e006c890 2024-01-21T03:56:35.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.859+0000 7fa8f17fa640 1 -- 172.21.15.7:0/646729171 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa8f4072860 con 0x7fa8e006c890 2024-01-21T03:56:35.859 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 7 to deep-scrub 2024-01-21T03:56:35.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 10 librados: watch_flush enter 2024-01-21T03:56:35.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 10 librados: watch_flush exit 2024-01-21T03:56:35.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 20 client.18077.objecter shutdown clearing up homeless session... 2024-01-21T03:56:35.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 10 client.18077.objecter successfully canceled tick 2024-01-21T03:56:35.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 1 -- 172.21.15.7:0/646729171 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fa8e006c890 msgr2=0x7fa8e006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 1 --2- 172.21.15.7:0/646729171 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fa8e006c890 0x7fa8e006ed50 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4137220 tx=0x7fa8e8009040 comp rx=0 tx=0).stop 2024-01-21T03:56:35.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 10 monclient: shutdown 2024-01-21T03:56:35.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 1 -- 172.21.15.7:0/646729171 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa8f40a8ab0 msgr2=0x7fa8f4135470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 1 --2- 172.21.15.7:0/646729171 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa8f40a8ab0 0x7fa8f4135470 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400e9b0 tx=0x7fa8e400ee70 comp rx=0 tx=0).stop 2024-01-21T03:56:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 1 -- 172.21.15.7:0/646729171 shutdown_connections 2024-01-21T03:56:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 1 --2- 172.21.15.7:0/646729171 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fa8e006c890 0x7fa8e006ed50 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 1 --2- 172.21.15.7:0/646729171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8f40b2310 0x7fa8f4132eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 1 --2- 172.21.15.7:0/646729171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8f40a8e90 0x7fa8f4132970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 1 --2- 172.21.15.7:0/646729171 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa8f40a8ab0 0x7fa8f4135470 unknown :-1 s=CLOSED pgs=3161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 1 -- 172.21.15.7:0/646729171 >> 172.21.15.7:0/646729171 conn(0x7fa8f4066f60 msgr2=0x7fa8f40b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 1 -- 172.21.15.7:0/646729171 shutdown_connections 2024-01-21T03:56:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 1 -- 172.21.15.7:0/646729171 wait complete. 2024-01-21T03:56:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.861+0000 7fa8fb076640 1 librados: shutdown 2024-01-21T03:56:35.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T03:56:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.976+0000 7f975942b640 10 monclient: get_monmap_and_config 2024-01-21T03:56:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.976+0000 7f975942b640 10 monclient: build_initial_monmap 2024-01-21T03:56:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.976+0000 7f975942b640 10 monclient: monmap: 2024-01-21T03:56:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:35.977151+0000 2024-01-21T03:56:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:35.977151+0000 2024-01-21T03:56:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:35.978 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.976+0000 7f975942b640 1 Processor -- start 2024-01-21T03:56:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.976+0000 7f975942b640 1 -- start start 2024-01-21T03:56:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.976+0000 7f975942b640 10 monclient: init 2024-01-21T03:56:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.977+0000 7f975942b640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.977+0000 7f975942b640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.977+0000 7f975942b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97540a8a40 addr v1:172.21.15.7:6789/0 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.977+0000 7f975942b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97540b21c0 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.977+0000 7f975942b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f975413f8b0 0x7f975414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.977+0000 7f975942b640 10 monclient(hunting): picked mon.noname-b con 0x7f975413f8b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.977+0000 7f975942b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97540aa9b0 con 0x7f97540a8a40 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.977+0000 7f975942b640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97540aac90 con 0x7f97540b21c0 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.977+0000 7f975942b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.977+0000 7f975942b640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97540aa7d0 con 0x7f975413f8b0 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.977+0000 7f975942b640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.977+0000 7f975942b640 10 monclient(hunting): authenticate will time out at 3542.293151s 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.977+0000 7f97537fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f975413f8b0 0x7f975414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.978+0000 7f9752ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f97540a8a40 0x7f9754145b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56438/0 (socket says 172.21.15.7:56438) 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.978+0000 7f9752ffd640 1 -- 172.21.15.7:0/1505695840 learned_addr learned my addr 172.21.15.7:0/1505695840 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.978+0000 7f97537fe640 10 monclient(hunting): get_auth_request con 0x7f975413f8b0 auth_method 0 2024-01-21T03:56:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.978+0000 7f97537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.978+0000 7f97537fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.978+0000 7f97537fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.978+0000 7f97537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.978+0000 7f97537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.978+0000 7f97537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.980+0000 7f97537fe640 10 monclient(hunting): handle_auth_done global_id 18080 payload 274 2024-01-21T03:56:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.980+0000 7f97537fe640 1 -- 172.21.15.7:0/1505695840 >> v1:172.21.15.177:6789/0 conn(0x7f97540b21c0 legacy=0x7f975413f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.980+0000 7f9751ffb640 1 -- 172.21.15.7:0/1505695840 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608820343 0 0) 0x7f97540aac90 con 0x7f97540b21c0 2024-01-21T03:56:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.980+0000 7f97537fe640 1 -- 172.21.15.7:0/1505695840 >> v1:172.21.15.7:6789/0 conn(0x7f97540a8a40 legacy=0x7f9754145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.980+0000 7f97537fe640 10 monclient: _finish_hunting 0 2024-01-21T03:56:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.980+0000 7f97537fe640 1 monclient: found mon.noname-b 2024-01-21T03:56:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.980+0000 7f97537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.980+0000 7f97537fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:56:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.980+0000 7f97537fe640 1 -- 172.21.15.7:0/1505695840 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97540aaf70 con 0x7f975413f8b0 2024-01-21T03:56:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.980+0000 7f9751ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-21T03:56:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.982+0000 7f97537fe640 1 --2- 172.21.15.7:0/1505695840 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f975413f8b0 0x7f975414d7f0 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 crypto rx=0x7f9744009dc0 tx=0x7f974402bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=493681d88e34a3c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f9751ffb640 1 -- 172.21.15.7:0/1505695840 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974402cd50 con 0x7f975413f8b0 2024-01-21T03:56:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f9751ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f9751ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f9751ffb640 10 monclient: dump: 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f9751ffb640 10 monclient: _finish_auth 0 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f9751ffb640 10 monclient: _check_auth_tickets 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f9751ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f9751ffb640 1 -- 172.21.15.7:0/1505695840 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9744034440 con 0x7f975413f8b0 2024-01-21T03:56:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f9751ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f9751ffb640 1 -- 172.21.15.7:0/1505695840 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97440347d0 con 0x7f975413f8b0 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f975942b640 5 monclient: authenticate success, global_id 18080 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f975942b640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f975942b640 4 set_mon_vals no callback set 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f9751ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f9751ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f9751ffb640 10 monclient: dump: 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:35.989 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f975942b640 10 monclient: shutdown 2024-01-21T03:56:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f975942b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f975942b640 1 -- 172.21.15.7:0/1505695840 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f975413f8b0 msgr2=0x7f975414d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f975942b640 1 --2- 172.21.15.7:0/1505695840 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f975413f8b0 0x7f975414d7f0 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 crypto rx=0x7f9744009dc0 tx=0x7f974402bfa0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f975942b640 1 -- 172.21.15.7:0/1505695840 shutdown_connections 2024-01-21T03:56:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f975942b640 1 --2- 172.21.15.7:0/1505695840 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f975413f8b0 0x7f975414d7f0 unknown :-1 s=CLOSED pgs=3162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f975942b640 1 -- 172.21.15.7:0/1505695840 >> 172.21.15.7:0/1505695840 conn(0x7f9754066f50 msgr2=0x7f97540b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f975942b640 1 -- 172.21.15.7:0/1505695840 shutdown_connections 2024-01-21T03:56:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.986+0000 7f975942b640 1 -- 172.21.15.7:0/1505695840 wait complete. 2024-01-21T03:56:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.987+0000 7f975942b640 10 monclient: build_initial_monmap 2024-01-21T03:56:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.987+0000 7f975942b640 10 monclient: monmap: 2024-01-21T03:56:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.987+0000 7f975942b640 1 librados: starting msgr at 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.987+0000 7f975942b640 1 librados: starting objecter 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.988+0000 7f975942b640 1 Processor -- start 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.988+0000 7f975942b640 1 -- start start 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.988+0000 7f975942b640 1 librados: setting wanted keys 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.988+0000 7f975942b640 1 librados: calling monclient init 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.988+0000 7f975942b640 10 monclient: init 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.988+0000 7f975942b640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.988+0000 7f975942b640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T03:56:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.988+0000 7f975942b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97540a8a40 0x7f97540b7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.988+0000 7f975942b640 10 monclient(hunting): picked mon.noname-c con 0x7f97540a8a40 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f975942b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97540b21c0 0x7f97541dac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f975942b640 10 monclient(hunting): picked mon.noname-a con 0x7f97540b21c0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f975942b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f975413f8b0 0x7f97541db190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f975942b640 10 monclient(hunting): picked mon.noname-b con 0x7f975413f8b0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f975942b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f975942b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97541426d0 con 0x7f97540b21c0 2024-01-21T03:56:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f975942b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f975942b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9754142360 con 0x7f975413f8b0 2024-01-21T03:56:35.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f975942b640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:35.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f975942b640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97540b8520 con 0x7f97540a8a40 2024-01-21T03:56:35.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f975942b640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f975942b640 10 monclient(hunting): authenticate will time out at 3542.304800s 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f97537fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f975413f8b0 0x7f97541db190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f97527fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97540b21c0 0x7f97541dac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f97537fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f975413f8b0 0x7f97541db190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:50056/0 (socket says 172.21.15.7:50056) 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f97537fe640 1 -- 172.21.15.7:0/3864849000 learned_addr learned my addr 172.21.15.7:0/3864849000 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f97537fe640 10 monclient(hunting): get_auth_request con 0x7f975413f8b0 auth_method 0 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f97537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f97537fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.989+0000 7f97537fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.990+0000 7f97527fc640 10 monclient(hunting): get_auth_request con 0x7f97540b21c0 auth_method 0 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.990+0000 7f97527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.990+0000 7f97527fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.990+0000 7f97527fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.990+0000 7f97537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.990+0000 7f97537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.990+0000 7f97537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.990+0000 7f97527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.990+0000 7f97527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.990+0000 7f97527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.991+0000 7f9752ffd640 1 --2- 172.21.15.7:0/3864849000 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97540a8a40 0x7f97540b7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.992+0000 7f9752ffd640 10 monclient(hunting): get_auth_request con 0x7f97540a8a40 auth_method 0 2024-01-21T03:56:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.992+0000 7f9752ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.992+0000 7f9752ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.992+0000 7f9752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.992+0000 7f9752ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.992+0000 7f9752ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.992+0000 7f9752ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.993+0000 7f97537fe640 10 monclient(hunting): handle_auth_done global_id 23242 payload 824 2024-01-21T03:56:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.993+0000 7f97537fe640 1 -- 172.21.15.7:0/3864849000 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97540a8a40 msgr2=0x7f97540b7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.993+0000 7f97537fe640 1 --2- 172.21.15.7:0/3864849000 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97540a8a40 0x7f97540b7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.993+0000 7f97537fe640 1 -- 172.21.15.7:0/3864849000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97540b21c0 msgr2=0x7f97541dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.993+0000 7f97537fe640 1 --2- 172.21.15.7:0/3864849000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97540b21c0 0x7f97541dac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.993+0000 7f97537fe640 10 monclient: _finish_hunting 0 2024-01-21T03:56:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.993+0000 7f97537fe640 1 monclient: found mon.noname-b 2024-01-21T03:56:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.993+0000 7f97537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.993+0000 7f97537fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:56:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.993+0000 7f97537fe640 1 -- 172.21.15.7:0/3864849000 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9744009a70 con 0x7f975413f8b0 2024-01-21T03:56:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.995+0000 7f97537fe640 1 --2- 172.21.15.7:0/3864849000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f975413f8b0 0x7f97541db190 secure :-1 s=READY pgs=5832 cs=0 l=1 rev1=1 crypto rx=0x7f974402bfe0 tx=0x7f9744004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.995+0000 7f973b7fe640 10 client.?.objecter ms_handle_connect 0x7f975413f8b0 2024-01-21T03:56:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.995+0000 7f973b7fe640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f973b7fe640 1 -- 172.21.15.7:0/3864849000 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9744040650 con 0x7f975413f8b0 2024-01-21T03:56:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f973b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f973b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f973b7fe640 10 monclient: dump: 2024-01-21T03:56:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f973b7fe640 10 monclient: _finish_auth 0 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f973b7fe640 10 monclient: _check_auth_tickets 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f973b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f973b7fe640 1 -- 172.21.15.7:0/3864849000 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9744034e90 con 0x7f975413f8b0 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f975942b640 5 monclient: authenticate success, global_id 23242 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f975942b640 10 monclient: _renew_subs 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f975942b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f975942b640 1 -- 172.21.15.7:0/3864849000 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97540b7050 con 0x7f975413f8b0 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f973b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f973b7fe640 1 -- 172.21.15.7:0/3864849000 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97440409d0 con 0x7f975413f8b0 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f973b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f973b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f973b7fe640 10 monclient: dump: 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f97517fa640 4 set_mon_vals no callback set 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f975942b640 10 client.23242.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f975942b640 10 monclient: _renew_subs 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f975942b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f975942b640 1 -- 172.21.15.7:0/3864849000 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97541db8b0 con 0x7f975413f8b0 2024-01-21T03:56:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:35.999+0000 7f975942b640 1 librados: init done 2024-01-21T03:56:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.000+0000 7f975942b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.000+0000 7f975942b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.000+0000 7f975942b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.000+0000 7f975942b640 1 -- 172.21.15.7:0/3864849000 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9754142520 con 0x7f975413f8b0 2024-01-21T03:56:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.019+0000 7f973b7fe640 1 -- 172.21.15.7:0/3864849000 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9744040b30 con 0x7f975413f8b0 2024-01-21T03:56:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.019+0000 7f973b7fe640 1 --2- 172.21.15.7:0/3864849000 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f974006c8b0 0x7f974006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.019+0000 7f9752ffd640 1 --2- 172.21.15.7:0/3864849000 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f974006c8b0 0x7f974006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.019+0000 7f9752ffd640 10 monclient: get_auth_request con 0x7f974006c8b0 auth_method 0 2024-01-21T03:56:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.020+0000 7f973b7fe640 10 client.23242.objecter ms_handle_connect 0x7f974006c8b0 2024-01-21T03:56:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.020+0000 7f9752ffd640 1 --2- 172.21.15.7:0/3864849000 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f974006c8b0 0x7f974006ed70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f974c007fd0 tx=0x7f974c007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:36.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.246+0000 7f973b7fe640 1 -- 172.21.15.7:0/3864849000 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7f974407b410 con 0x7f975413f8b0 2024-01-21T03:56:36.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.246+0000 7f973b7fe640 10 client.23242.objecter ms_dispatch 0x7f9754000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:36.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.246+0000 7f973b7fe640 3 client.23242.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:36.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.246+0000 7f973b7fe640 3 client.23242.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:36.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.247+0000 7f973b7fe640 20 client.23242.objecter dump_active .. 0 homeless 2024-01-21T03:56:36.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.257+0000 7f973b7fe640 1 -- 172.21.15.7:0/3864849000 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9744077750 con 0x7f975413f8b0 2024-01-21T03:56:36.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.257+0000 7f973b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:36.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.257+0000 7f973b7fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:36.372 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:50.686 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:36.425 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:50.740 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:36.441 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:50.701 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:36.445 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:50.704 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:36.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.449+0000 7f975942b640 1 -- 172.21.15.7:0/3864849000 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97541461f0 con 0x7f974006c8b0 2024-01-21T03:56:36.454 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:37:50.768 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.454+0000 7f973b7fe640 1 -- 172.21.15.7:0/3864849000 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151279 (secure 0 0 0) 0x7f97541461f0 con 0x7f974006c8b0 2024-01-21T03:56:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:56:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-01-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.456+0000 7f975942b640 10 librados: watch_flush enter 2024-01-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 10 librados: watch_flush exit 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 20 client.23242.objecter shutdown clearing up homeless session... 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 10 client.23242.objecter successfully canceled tick 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 1 -- 172.21.15.7:0/3864849000 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f974006c8b0 msgr2=0x7f974006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 1 --2- 172.21.15.7:0/3864849000 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f974006c8b0 0x7f974006ed70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f974c007fd0 tx=0x7f974c007e80 comp rx=0 tx=0).stop 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 10 monclient: shutdown 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 1 -- 172.21.15.7:0/3864849000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f975413f8b0 msgr2=0x7f97541db190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 1 --2- 172.21.15.7:0/3864849000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f975413f8b0 0x7f97541db190 secure :-1 s=READY pgs=5832 cs=0 l=1 rev1=1 crypto rx=0x7f974402bfe0 tx=0x7f9744004a40 comp rx=0 tx=0).stop 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 1 -- 172.21.15.7:0/3864849000 shutdown_connections 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 1 --2- 172.21.15.7:0/3864849000 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f974006c8b0 0x7f974006ed70 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 1 --2- 172.21.15.7:0/3864849000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f975413f8b0 0x7f97541db190 unknown :-1 s=CLOSED pgs=5832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 1 --2- 172.21.15.7:0/3864849000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97540b21c0 0x7f97541dac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 1 --2- 172.21.15.7:0/3864849000 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97540a8a40 0x7f97540b7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 1 -- 172.21.15.7:0/3864849000 >> 172.21.15.7:0/3864849000 conn(0x7f9754066f50 msgr2=0x7f975413cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 1 -- 172.21.15.7:0/3864849000 shutdown_connections 2024-01-21T03:56:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 1 -- 172.21.15.7:0/3864849000 wait complete. 2024-01-21T03:56:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:36.457+0000 7f975942b640 1 librados: shutdown 2024-01-21T03:56:36.463 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:37:50.723 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:36.469 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":2403,"stamp":"2024-01-21T03:56:36.234628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":600960,"kb_used_data":40600,"kb_used_omap":12,"kb_used_meta":560243,"kb_avail":754373760,"statfs":{"total":773094113280,"available":772478730240,"internally_reserved":0,"allocated":41574400,"data_stored":19456474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":573689393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.143184"},"pg_stats":[{"pgid":"14.1e","version":"0'0","reported_seq":1256,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.135756+0000","last_change":"2024-01-21T03:21:15.540562+0000","last_active":"2024-01-21T03:56:25.135756+0000","last_peered":"2024-01-21T03:56:25.135756+0000","last_clean":"2024-01-21T03:56:25.135756+0000","last_became_active":"2024-01-21T03:21:15.289479+0000","last_became_peered":"2024-01-21T03:21:15.289479+0000","last_unstale":"2024-01-21T03:56:25.135756+0000","last_undegraded":"2024-01-21T03:56:25.135756+0000","last_fullsized":"2024-01-21T03:56:25.135756+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1214,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.346465+0000","last_change":"2024-01-21T03:21:15.879415+0000","last_active":"2024-01-21T03:56:25.346465+0000","last_peered":"2024-01-21T03:56:25.346465+0000","last_clean":"2024-01-21T03:56:25.346465+0000","last_became_active":"2024-01-21T03:21:15.864718+0000","last_became_peered":"2024-01-21T03:21:15.864718+0000","last_unstale":"2024-01-21T03:56:25.346465+0000","last_undegraded":"2024-01-21T03:56:25.346465+0000","last_fullsized":"2024-01-21T03:56:25.346465+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:26:29.398180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1235,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.317220+0000","last_change":"2024-01-21T03:21:09.350719+0000","last_active":"2024-01-21T03:56:24.317220+0000","last_peered":"2024-01-21T03:56:24.317220+0000","last_clean":"2024-01-21T03:56:24.317220+0000","last_became_active":"2024-01-21T03:21:09.333873+0000","last_became_peered":"2024-01-21T03:21:09.333873+0000","last_unstale":"2024-01-21T03:56:24.317220+0000","last_undegraded":"2024-01-21T03:56:24.317220+0000","last_fullsized":"2024-01-21T03:56:24.317220+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:44:56.009576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1232,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.143419+0000","last_change":"2024-01-21T03:21:13.545831+0000","last_active":"2024-01-21T03:56:24.143419+0000","last_peered":"2024-01-21T03:56:24.143419+0000","last_clean":"2024-01-21T03:56:24.143419+0000","last_became_active":"2024-01-21T03:21:13.539461+0000","last_became_peered":"2024-01-21T03:21:13.539461+0000","last_unstale":"2024-01-21T03:56:24.143419+0000","last_undegraded":"2024-01-21T03:56:24.143419+0000","last_fullsized":"2024-01-21T03:56:24.143419+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:55:11.327991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1212,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.284435+0000","last_change":"2024-01-21T03:21:31.051184+0000","last_active":"2024-01-21T03:56:25.284435+0000","last_peered":"2024-01-21T03:56:25.284435+0000","last_clean":"2024-01-21T03:56:25.284435+0000","last_became_active":"2024-01-21T03:21:30.818768+0000","last_became_peered":"2024-01-21T03:21:30.818768+0000","last_unstale":"2024-01-21T03:56:25.284435+0000","last_undegraded":"2024-01-21T03:56:25.284435+0000","last_fullsized":"2024-01-21T03:56:25.284435+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1234,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.168747+0000","last_change":"2024-01-21T03:21:13.851032+0000","last_active":"2024-01-21T03:56:24.168747+0000","last_peered":"2024-01-21T03:56:24.168747+0000","last_clean":"2024-01-21T03:56:24.168747+0000","last_became_active":"2024-01-21T03:21:13.842398+0000","last_became_peered":"2024-01-21T03:21:13.842398+0000","last_unstale":"2024-01-21T03:56:24.168747+0000","last_undegraded":"2024-01-21T03:56:24.168747+0000","last_fullsized":"2024-01-21T03:56:24.168747+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:27:03.441063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1235,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.184161+0000","last_change":"2024-01-21T03:21:34.296238+0000","last_active":"2024-01-21T03:56:24.184161+0000","last_peered":"2024-01-21T03:56:24.184161+0000","last_clean":"2024-01-21T03:56:24.184161+0000","last_became_active":"2024-01-21T03:21:34.290896+0000","last_became_peered":"2024-01-21T03:21:34.290896+0000","last_unstale":"2024-01-21T03:56:24.184161+0000","last_undegraded":"2024-01-21T03:56:24.184161+0000","last_fullsized":"2024-01-21T03:56:24.184161+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:52:43.200440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1","version":"60'7","reported_seq":1228,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.327209+0000","last_change":"2024-01-21T03:21:35.777407+0000","last_active":"2024-01-21T03:56:25.327209+0000","last_peered":"2024-01-21T03:56:25.327209+0000","last_clean":"2024-01-21T03:56:25.327209+0000","last_became_active":"2024-01-21T03:21:35.763549+0000","last_became_peered":"2024-01-21T03:21:35.763549+0000","last_unstale":"2024-01-21T03:56:25.327209+0000","last_undegraded":"2024-01-21T03:56:25.327209+0000","last_fullsized":"2024-01-21T03:56:25.327209+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:56:17.661965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1033,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:31.457626+0000","last_change":"2024-01-21T03:56:31.458427+0000","last_active":"2024-01-21T03:56:31.457626+0000","last_peered":"2024-01-21T03:56:31.457626+0000","last_clean":"2024-01-21T03:56:31.457626+0000","last_became_active":"2024-01-21T03:26:06.758343+0000","last_became_peered":"2024-01-21T03:26:06.758343+0000","last_unstale":"2024-01-21T03:56:31.457626+0000","last_undegraded":"2024-01-21T03:56:31.457626+0000","last_fullsized":"2024-01-21T03:56:31.457626+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:31.456183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:31.456183+0000","last_clean_scrub_stamp":"2024-01-21T03:56:31.456183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:29:15.558115+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1231,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.370616+0000","last_change":"2024-01-21T03:21:15.810535+0000","last_active":"2024-01-21T03:56:25.370616+0000","last_peered":"2024-01-21T03:56:25.370616+0000","last_clean":"2024-01-21T03:56:25.370616+0000","last_became_active":"2024-01-21T03:21:15.779865+0000","last_became_peered":"2024-01-21T03:21:15.779865+0000","last_unstale":"2024-01-21T03:56:25.370616+0000","last_undegraded":"2024-01-21T03:56:25.370616+0000","last_fullsized":"2024-01-21T03:56:25.370616+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1212,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.297079+0000","last_change":"2024-01-21T03:21:22.661192+0000","last_active":"2024-01-21T03:56:25.297079+0000","last_peered":"2024-01-21T03:56:25.297079+0000","last_clean":"2024-01-21T03:56:25.297079+0000","last_became_active":"2024-01-21T03:21:22.647696+0000","last_became_peered":"2024-01-21T03:21:22.647696+0000","last_unstale":"2024-01-21T03:56:25.297079+0000","last_undegraded":"2024-01-21T03:56:25.297079+0000","last_fullsized":"2024-01-21T03:56:25.297079+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1222,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.330587+0000","last_change":"2024-01-21T03:21:26.231500+0000","last_active":"2024-01-21T03:56:24.330587+0000","last_peered":"2024-01-21T03:56:24.330587+0000","last_clean":"2024-01-21T03:56:24.330587+0000","last_became_active":"2024-01-21T03:21:26.223079+0000","last_became_peered":"2024-01-21T03:21:26.223079+0000","last_unstale":"2024-01-21T03:56:24.330587+0000","last_undegraded":"2024-01-21T03:56:24.330587+0000","last_fullsized":"2024-01-21T03:56:24.330587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:36:33.965862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1237,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.372831+0000","last_change":"2024-01-21T03:26:06.699602+0000","last_active":"2024-01-21T03:56:25.372831+0000","last_peered":"2024-01-21T03:56:25.372831+0000","last_clean":"2024-01-21T03:56:25.372831+0000","last_became_active":"2024-01-21T03:26:06.693281+0000","last_became_peered":"2024-01-21T03:26:06.693281+0000","last_unstale":"2024-01-21T03:56:25.372831+0000","last_undegraded":"2024-01-21T03:56:25.372831+0000","last_fullsized":"2024-01-21T03:56:25.372831+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1294'78","reported_seq":1272,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:30.986195+0000","last_change":"2024-01-21T03:56:30.987756+0000","last_active":"2024-01-21T03:56:30.986195+0000","last_peered":"2024-01-21T03:56:30.986195+0000","last_clean":"2024-01-21T03:56:30.986195+0000","last_became_active":"2024-01-21T03:19:58.060095+0000","last_became_peered":"2024-01-21T03:19:58.060095+0000","last_unstale":"2024-01-21T03:56:30.986195+0000","last_undegraded":"2024-01-21T03:56:30.986195+0000","last_fullsized":"2024-01-21T03:56:30.986195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"1294'78","last_scrub_stamp":"2024-01-21T03:56:30.981858+0000","last_deep_scrub":"1294'78","last_deep_scrub_stamp":"2024-01-21T03:56:30.981858+0000","last_clean_scrub_stamp":"2024-01-21T03:56:30.981858+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:38:53.709518+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1262,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.133027+0000","last_change":"2024-01-21T03:20:02.295222+0000","last_active":"2024-01-21T03:56:25.133027+0000","last_peered":"2024-01-21T03:56:25.133027+0000","last_clean":"2024-01-21T03:56:25.133027+0000","last_became_active":"2024-01-21T03:20:02.289915+0000","last_became_peered":"2024-01-21T03:20:02.289915+0000","last_unstale":"2024-01-21T03:56:25.133027+0000","last_undegraded":"2024-01-21T03:56:25.133027+0000","last_fullsized":"2024-01-21T03:56:25.133027+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1222,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.304766+0000","last_change":"2024-01-21T03:21:14.853643+0000","last_active":"2024-01-21T03:56:24.304766+0000","last_peered":"2024-01-21T03:56:24.304766+0000","last_clean":"2024-01-21T03:56:24.304766+0000","last_became_active":"2024-01-21T03:21:14.829810+0000","last_became_peered":"2024-01-21T03:21:14.829810+0000","last_unstale":"2024-01-21T03:56:24.304766+0000","last_undegraded":"2024-01-21T03:56:24.304766+0000","last_fullsized":"2024-01-21T03:56:24.304766+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:30:12.597754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1271,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:33.190367+0000","last_change":"2024-01-21T03:56:33.191175+0000","last_active":"2024-01-21T03:56:33.190367+0000","last_peered":"2024-01-21T03:56:33.190367+0000","last_clean":"2024-01-21T03:56:33.190367+0000","last_became_active":"2024-01-21T03:20:01.784281+0000","last_became_peered":"2024-01-21T03:20:01.784281+0000","last_unstale":"2024-01-21T03:56:33.190367+0000","last_undegraded":"2024-01-21T03:56:33.190367+0000","last_fullsized":"2024-01-21T03:56:33.190367+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:33.188920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:33.188920+0000","last_clean_scrub_stamp":"2024-01-21T03:56:33.188920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:14:38.519289+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1256,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.127049+0000","last_change":"2024-01-21T03:21:26.074817+0000","last_active":"2024-01-21T03:56:25.127049+0000","last_peered":"2024-01-21T03:56:25.127049+0000","last_clean":"2024-01-21T03:56:25.127049+0000","last_became_active":"2024-01-21T03:21:26.057996+0000","last_became_peered":"2024-01-21T03:21:26.057996+0000","last_unstale":"2024-01-21T03:56:25.127049+0000","last_undegraded":"2024-01-21T03:56:25.127049+0000","last_fullsized":"2024-01-21T03:56:25.127049+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1219,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.297677+0000","last_change":"2024-01-21T03:20:01.846748+0000","last_active":"2024-01-21T03:56:25.297677+0000","last_peered":"2024-01-21T03:56:25.297677+0000","last_clean":"2024-01-21T03:56:25.297677+0000","last_became_active":"2024-01-21T03:20:01.818592+0000","last_became_peered":"2024-01-21T03:20:01.818592+0000","last_unstale":"2024-01-21T03:56:25.297677+0000","last_undegraded":"2024-01-21T03:56:25.297677+0000","last_fullsized":"2024-01-21T03:56:25.297677+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:08:09.861694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1231,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.367118+0000","last_change":"2024-01-21T03:22:06.056192+0000","last_active":"2024-01-21T03:56:25.367118+0000","last_peered":"2024-01-21T03:56:25.367118+0000","last_clean":"2024-01-21T03:56:25.367118+0000","last_became_active":"2024-01-21T03:22:06.040269+0000","last_became_peered":"2024-01-21T03:22:06.040269+0000","last_unstale":"2024-01-21T03:56:25.367118+0000","last_undegraded":"2024-01-21T03:56:25.367118+0000","last_fullsized":"2024-01-21T03:56:25.367118+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1262,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.116124+0000","last_change":"2024-01-21T03:20:02.210632+0000","last_active":"2024-01-21T03:56:25.116124+0000","last_peered":"2024-01-21T03:56:25.116124+0000","last_clean":"2024-01-21T03:56:25.116124+0000","last_became_active":"2024-01-21T03:20:02.201314+0000","last_became_peered":"2024-01-21T03:20:02.201314+0000","last_unstale":"2024-01-21T03:56:25.116124+0000","last_undegraded":"2024-01-21T03:56:25.116124+0000","last_fullsized":"2024-01-21T03:56:25.116124+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1234,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.182049+0000","last_change":"2024-01-21T03:21:34.314505+0000","last_active":"2024-01-21T03:56:24.182049+0000","last_peered":"2024-01-21T03:56:24.182049+0000","last_clean":"2024-01-21T03:56:24.182049+0000","last_became_active":"2024-01-21T03:21:34.307508+0000","last_became_peered":"2024-01-21T03:21:34.307508+0000","last_unstale":"2024-01-21T03:56:24.182049+0000","last_undegraded":"2024-01-21T03:56:24.182049+0000","last_fullsized":"2024-01-21T03:56:24.182049+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:38:13.727492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1228,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.306396+0000","last_change":"2024-01-21T03:20:02.214415+0000","last_active":"2024-01-21T03:56:24.306396+0000","last_peered":"2024-01-21T03:56:24.306396+0000","last_clean":"2024-01-21T03:56:24.306396+0000","last_became_active":"2024-01-21T03:20:02.207814+0000","last_became_peered":"2024-01-21T03:20:02.207814+0000","last_unstale":"2024-01-21T03:56:24.306396+0000","last_undegraded":"2024-01-21T03:56:24.306396+0000","last_fullsized":"2024-01-21T03:56:24.306396+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:12:28.359349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1224,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:30.648133+0000","last_change":"2024-01-21T03:56:30.652891+0000","last_active":"2024-01-21T03:56:30.648133+0000","last_peered":"2024-01-21T03:56:30.648133+0000","last_clean":"2024-01-21T03:56:30.648133+0000","last_became_active":"2024-01-21T03:21:23.767027+0000","last_became_peered":"2024-01-21T03:21:23.767027+0000","last_unstale":"2024-01-21T03:56:30.648133+0000","last_undegraded":"2024-01-21T03:56:30.648133+0000","last_fullsized":"2024-01-21T03:56:30.648133+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:30.644030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:30.644030+0000","last_clean_scrub_stamp":"2024-01-21T03:56:30.644030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:43:36.949817+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1270,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:34.201029+0000","last_change":"2024-01-21T03:56:34.201835+0000","last_active":"2024-01-21T03:56:34.201029+0000","last_peered":"2024-01-21T03:56:34.201029+0000","last_clean":"2024-01-21T03:56:34.201029+0000","last_became_active":"2024-01-21T03:20:01.872724+0000","last_became_peered":"2024-01-21T03:20:01.872724+0000","last_unstale":"2024-01-21T03:56:34.201029+0000","last_undegraded":"2024-01-21T03:56:34.201029+0000","last_fullsized":"2024-01-21T03:56:34.201029+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:34.199531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:34.199531+0000","last_clean_scrub_stamp":"2024-01-21T03:56:34.199531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:24:00.875702+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1212,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.286096+0000","last_change":"2024-01-21T03:21:55.470689+0000","last_active":"2024-01-21T03:56:25.286096+0000","last_peered":"2024-01-21T03:56:25.286096+0000","last_clean":"2024-01-21T03:56:25.286096+0000","last_became_active":"2024-01-21T03:21:55.460120+0000","last_became_peered":"2024-01-21T03:21:55.460120+0000","last_unstale":"2024-01-21T03:56:25.286096+0000","last_undegraded":"2024-01-21T03:56:25.286096+0000","last_fullsized":"2024-01-21T03:56:25.286096+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1238,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.205311+0000","last_change":"2024-01-21T03:20:02.260437+0000","last_active":"2024-01-21T03:56:24.205311+0000","last_peered":"2024-01-21T03:56:24.205311+0000","last_clean":"2024-01-21T03:56:24.205311+0000","last_became_active":"2024-01-21T03:20:02.234092+0000","last_became_peered":"2024-01-21T03:20:02.234092+0000","last_unstale":"2024-01-21T03:56:24.205311+0000","last_undegraded":"2024-01-21T03:56:24.205311+0000","last_fullsized":"2024-01-21T03:56:24.205311+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:32:42.605439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1236,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.320229+0000","last_change":"2024-01-21T03:21:09.536570+0000","last_active":"2024-01-21T03:56:24.320229+0000","last_peered":"2024-01-21T03:56:24.320229+0000","last_clean":"2024-01-21T03:56:24.320229+0000","last_became_active":"2024-01-21T03:21:09.531293+0000","last_became_peered":"2024-01-21T03:21:09.531293+0000","last_unstale":"2024-01-21T03:56:24.320229+0000","last_undegraded":"2024-01-21T03:56:24.320229+0000","last_fullsized":"2024-01-21T03:56:24.320229+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:01:17.017524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1258,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.138665+0000","last_change":"2024-01-21T03:20:02.233166+0000","last_active":"2024-01-21T03:56:25.138665+0000","last_peered":"2024-01-21T03:56:25.138665+0000","last_clean":"2024-01-21T03:56:25.138665+0000","last_became_active":"2024-01-21T03:20:02.226975+0000","last_became_peered":"2024-01-21T03:20:02.226975+0000","last_unstale":"2024-01-21T03:56:25.138665+0000","last_undegraded":"2024-01-21T03:56:25.138665+0000","last_fullsized":"2024-01-21T03:56:25.138665+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:19:59.904069+0000","last_clean_scrub_stamp":"2024-01-21T03:19:59.904069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1255,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.114359+0000","last_change":"2024-01-21T03:21:36.938379+0000","last_active":"2024-01-21T03:56:25.114359+0000","last_peered":"2024-01-21T03:56:25.114359+0000","last_clean":"2024-01-21T03:56:25.114359+0000","last_became_active":"2024-01-21T03:21:36.929227+0000","last_became_peered":"2024-01-21T03:21:36.929227+0000","last_unstale":"2024-01-21T03:56:25.114359+0000","last_undegraded":"2024-01-21T03:56:25.114359+0000","last_fullsized":"2024-01-21T03:56:25.114359+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1222,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.260384+0000","last_change":"2024-01-21T03:21:13.975490+0000","last_active":"2024-01-21T03:56:24.260384+0000","last_peered":"2024-01-21T03:56:24.260384+0000","last_clean":"2024-01-21T03:56:24.260384+0000","last_became_active":"2024-01-21T03:21:13.969066+0000","last_became_peered":"2024-01-21T03:21:13.969066+0000","last_unstale":"2024-01-21T03:56:24.260384+0000","last_undegraded":"2024-01-21T03:56:24.260384+0000","last_fullsized":"2024-01-21T03:56:24.260384+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:53:18.899854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1235,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.321807+0000","last_change":"2024-01-21T03:21:22.338167+0000","last_active":"2024-01-21T03:56:24.321807+0000","last_peered":"2024-01-21T03:56:24.321807+0000","last_clean":"2024-01-21T03:56:24.321807+0000","last_became_active":"2024-01-21T03:21:22.319120+0000","last_became_peered":"2024-01-21T03:21:22.319120+0000","last_unstale":"2024-01-21T03:56:24.321807+0000","last_undegraded":"2024-01-21T03:56:24.321807+0000","last_fullsized":"2024-01-21T03:56:24.321807+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:35:43.974713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1232,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.139273+0000","last_change":"2024-01-21T03:21:17.587236+0000","last_active":"2024-01-21T03:56:24.139273+0000","last_peered":"2024-01-21T03:56:24.139273+0000","last_clean":"2024-01-21T03:56:24.139273+0000","last_became_active":"2024-01-21T03:21:17.582134+0000","last_became_peered":"2024-01-21T03:21:17.582134+0000","last_unstale":"2024-01-21T03:56:24.139273+0000","last_undegraded":"2024-01-21T03:56:24.139273+0000","last_fullsized":"2024-01-21T03:56:24.139273+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:14:12.050155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1255,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.128663+0000","last_change":"2024-01-21T03:21:40.581757+0000","last_active":"2024-01-21T03:56:25.128663+0000","last_peered":"2024-01-21T03:56:25.128663+0000","last_clean":"2024-01-21T03:56:25.128663+0000","last_became_active":"2024-01-21T03:21:40.568805+0000","last_became_peered":"2024-01-21T03:21:40.568805+0000","last_unstale":"2024-01-21T03:56:25.128663+0000","last_undegraded":"2024-01-21T03:56:25.128663+0000","last_fullsized":"2024-01-21T03:56:25.128663+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1231,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.378116+0000","last_change":"2024-01-21T03:22:06.157185+0000","last_active":"2024-01-21T03:56:25.378116+0000","last_peered":"2024-01-21T03:56:25.378116+0000","last_clean":"2024-01-21T03:56:25.378116+0000","last_became_active":"2024-01-21T03:22:06.149457+0000","last_became_peered":"2024-01-21T03:22:06.149457+0000","last_unstale":"2024-01-21T03:56:25.378116+0000","last_undegraded":"2024-01-21T03:56:25.378116+0000","last_fullsized":"2024-01-21T03:56:25.378116+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1232,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.224082+0000","last_change":"2024-01-21T03:21:13.379783+0000","last_active":"2024-01-21T03:56:24.224082+0000","last_peered":"2024-01-21T03:56:24.224082+0000","last_clean":"2024-01-21T03:56:24.224082+0000","last_became_active":"2024-01-21T03:21:13.373828+0000","last_became_peered":"2024-01-21T03:21:13.373828+0000","last_unstale":"2024-01-21T03:56:24.224082+0000","last_undegraded":"2024-01-21T03:56:24.224082+0000","last_fullsized":"2024-01-21T03:56:24.224082+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:54:16.898824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1233,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.330012+0000","last_change":"2024-01-21T03:21:11.007665+0000","last_active":"2024-01-21T03:56:24.330012+0000","last_peered":"2024-01-21T03:56:24.330012+0000","last_clean":"2024-01-21T03:56:24.330012+0000","last_became_active":"2024-01-21T03:21:10.999264+0000","last_became_peered":"2024-01-21T03:21:10.999264+0000","last_unstale":"2024-01-21T03:56:24.330012+0000","last_undegraded":"2024-01-21T03:56:24.330012+0000","last_fullsized":"2024-01-21T03:56:24.330012+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:43:27.507683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1231,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:25.352006+0000","last_change":"2024-01-21T03:21:15.387706+0000","last_active":"2024-01-21T03:56:25.352006+0000","last_peered":"2024-01-21T03:56:25.352006+0000","last_clean":"2024-01-21T03:56:25.352006+0000","last_became_active":"2024-01-21T03:21:15.068960+0000","last_became_peered":"2024-01-21T03:21:15.068960+0000","last_unstale":"2024-01-21T03:56:25.352006+0000","last_undegraded":"2024-01-21T03:56:25.352006+0000","last_fullsized":"2024-01-21T03:56:25.352006+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1236,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.305574+0000","last_change":"2024-01-21T03:21:33.019724+0000","last_active":"2024-01-21T03:56:24.305574+0000","last_peered":"2024-01-21T03:56:24.305574+0000","last_clean":"2024-01-21T03:56:24.305574+0000","last_became_active":"2024-01-21T03:21:32.997061+0000","last_became_peered":"2024-01-21T03:21:32.997061+0000","last_unstale":"2024-01-21T03:56:24.305574+0000","last_undegraded":"2024-01-21T03:56:24.305574+0000","last_fullsized":"2024-01-21T03:56:24.305574+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:52:53.090792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1222,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.299999+0000","last_change":"2024-01-21T03:21:14.774673+0000","last_active":"2024-01-21T03:56:24.299999+0000","last_peered":"2024-01-21T03:56:24.299999+0000","last_clean":"2024-01-21T03:56:24.299999+0000","last_became_active":"2024-01-21T03:21:14.761665+0000","last_became_peered":"2024-01-21T03:21:14.761665+0000","last_unstale":"2024-01-21T03:56:24.299999+0000","last_undegraded":"2024-01-21T03:56:24.299999+0000","last_fullsized":"2024-01-21T03:56:24.299999+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:09:47.334671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1236,"reported_epoch":1293,"state":"active+clean","last_fresh":"2024-01-21T03:56:24.313486+0000","last_change":"2024-01-21T03:21:34.181963+0000","last_active":"2024-01-21T03:56:24.313486+0000","last_peered":"2024-01-21T03:56:24.313486+0000","last_clean":"2024-01-21T03:56:24.313486+0000","last_became_active":"2024-01-21T03:21:34.168272+0000","last_became_peered":"2024-01-21T03:21:34.168272+0000","last_unstale":"2024-01-21T03:56:24.313486+0000","last_undegraded":"2024-01-21T03:56:24.313486+0000","last_fullsized":"2024-01-21T03:56:24.313486+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:21:02.392695+0000","last_clean_scrub_stamp":"2024-01-21T03:21:02.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:16:18.395527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280950,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76408,"kb_used_data":4976,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295432,"statfs":{"total":96636764160,"available":96558522368,"internally_reserved":0,"allocated":5095424,"data_stored":2332598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":9.3070000000000004,"5min":22.673999999999999,"15min":23.361000000000001},"min":{"1min":5.9930000000000003,"5min":4.7569999999999997,"15min":4.2290000000000001},"max":{"1min":15.645,"5min":221.726,"15min":236.40199999999999},"last":9.1880000000000006},{"interface":"front","average":{"1min":12.923,"5min":28.297999999999998,"15min":22.507000000000001},"min":{"1min":7.3280000000000003,"5min":6.5869999999999997,"15min":4.423},"max":{"1min":21.279,"5min":315.84899999999999,"15min":317.16500000000002},"last":14.949999999999999}]},{"osd":1,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":12.68,"5min":24.010999999999999,"15min":26.009},"min":{"1min":7.8280000000000003,"5min":7.8280000000000003,"15min":6.7649999999999997},"max":{"1min":19.681999999999999,"5min":210.08799999999999,"15min":688.34500000000003},"last":15.631},{"interface":"front","average":{"1min":15.066000000000001,"5min":30.968,"15min":31.401},"min":{"1min":10.843999999999999,"5min":10.49,"15min":9.6310000000000002},"max":{"1min":23.169,"5min":288.46600000000001,"15min":686.94299999999998},"last":16.876000000000001}]},{"osd":2,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":9.9570000000000007,"5min":59.222999999999999,"15min":34.966999999999999},"min":{"1min":6.0730000000000004,"5min":6.0730000000000004,"15min":5.75},"max":{"1min":31.401,"5min":3500.2379999999998,"15min":3500.2379999999998},"last":8.2899999999999991},{"interface":"front","average":{"1min":14.031000000000001,"5min":37.201000000000001,"15min":29.867000000000001},"min":{"1min":7.9669999999999996,"5min":7.9669999999999996,"15min":7.4710000000000001},"max":{"1min":25.942,"5min":919.58199999999999,"15min":919.58199999999999},"last":13.199999999999999}]},{"osd":4,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":14.010999999999999,"5min":26.649000000000001,"15min":26.817},"min":{"1min":10.026,"5min":9.1489999999999991,"15min":8.9909999999999997},"max":{"1min":18.762,"5min":241.95599999999999,"15min":297.46600000000001},"last":16.273},{"interface":"front","average":{"1min":18.001999999999999,"5min":31.356999999999999,"15min":31.646999999999998},"min":{"1min":12.919,"5min":12.346,"15min":9.673},"max":{"1min":23.687000000000001,"5min":292.25400000000002,"15min":340.73500000000001},"last":18.399999999999999}]},{"osd":5,"last update":"Sun Jan 21 03:56:18 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":24.042999999999999,"15min":24.727},"min":{"1min":7.3680000000000003,"5min":7.3680000000000003,"15min":6.1989999999999998},"max":{"1min":22.300000000000001,"5min":214.58500000000001,"15min":362.22899999999998},"last":12.164},{"interface":"front","average":{"1min":13.831,"5min":26.408000000000001,"15min":27.535},"min":{"1min":8.9079999999999995,"5min":6.5439999999999996,"15min":6.5439999999999996},"max":{"1min":21.576000000000001,"5min":339.19499999999999,"15min":339.19499999999999},"last":11.284000000000001}]},{"osd":6,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":18.814,"5min":31.257000000000001,"15min":30.303999999999998},"min":{"1min":11.645,"5min":11.206,"15min":7.1719999999999997},"max":{"1min":42.293999999999997,"5min":211.822,"15min":572.20899999999995},"last":19.245000000000001},{"interface":"front","average":{"1min":21.713000000000001,"5min":36.789999999999999,"15min":34.954999999999998},"min":{"1min":13.535,"5min":13.535,"15min":9.7240000000000002},"max":{"1min":45.457999999999998,"5min":215.315,"15min":376.66399999999999},"last":20.913}]},{"osd":7,"last update":"Sun Jan 21 03:56:18 2024","interfaces":[{"interface":"back","average":{"1min":15.071999999999999,"5min":23.681999999999999,"15min":23.119},"min":{"1min":9.3960000000000008,"5min":8.1329999999999991,"15min":5.9169999999999998},"max":{"1min":30.167999999999999,"5min":213.73500000000001,"15min":274.91000000000003},"last":9.9160000000000004},{"interface":"front","average":{"1min":21.024000000000001,"5min":32.463999999999999,"15min":32.999000000000002},"min":{"1min":14.153,"5min":11.337,"15min":11.337},"max":{"1min":31.466000000000001,"5min":234.46299999999999,"15min":339.613},"last":20.053999999999998}]}]},{"osd":2,"up_from":21,"seq":90194313658,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68096,"kb_used_data":4976,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303744,"statfs":{"total":96636764160,"available":96567033856,"internally_reserved":0,"allocated":5095424,"data_stored":2332598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:55:51 2024","interfaces":[{"interface":"back","average":{"1min":11.087,"5min":20.286999999999999,"15min":17.802},"min":{"1min":4.9820000000000002,"5min":4.9820000000000002,"15min":4.5460000000000003},"max":{"1min":23.617000000000001,"5min":335.38200000000001,"15min":335.38200000000001},"last":8.25},{"interface":"front","average":{"1min":9.5939999999999994,"5min":20.925999999999998,"15min":17.704000000000001},"min":{"1min":5.484,"5min":5.484,"15min":4.4580000000000002},"max":{"1min":22.068999999999999,"5min":343.05599999999998,"15min":343.05599999999998},"last":9.8119999999999994}]},{"osd":1,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":16.661999999999999,"5min":32.128,"15min":25.600999999999999},"min":{"1min":7.641,"5min":5.5960000000000001,"15min":5.2190000000000003},"max":{"1min":50.104999999999997,"5min":341.34399999999999,"15min":504.58100000000002},"last":11.878},{"interface":"front","average":{"1min":14.928000000000001,"5min":35.445,"15min":27.994},"min":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":5.7350000000000003},"max":{"1min":32.649000000000001,"5min":674.13400000000001,"15min":674.13400000000001},"last":13.172000000000001}]},{"osd":3,"last update":"Sun Jan 21 03:55:37 2024","interfaces":[{"interface":"back","average":{"1min":14.738,"5min":22.449999999999999,"15min":20.140999999999998},"min":{"1min":7.9509999999999996,"5min":6.9820000000000002,"15min":6.3659999999999997},"max":{"1min":45.518999999999998,"5min":263.31400000000002,"15min":263.31400000000002},"last":15.137},{"interface":"front","average":{"1min":17.158999999999999,"5min":26.888000000000002,"15min":22.359999999999999},"min":{"1min":9.6389999999999993,"5min":8.1180000000000003,"15min":8.1180000000000003},"max":{"1min":50.167000000000002,"5min":683.75800000000004,"15min":683.75800000000004},"last":14.545}]},{"osd":4,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":22.283000000000001,"5min":29.207999999999998,"15min":23.219999999999999},"min":{"1min":10.741,"5min":8.827,"15min":8.0609999999999999},"max":{"1min":68.728999999999999,"5min":687.15800000000002,"15min":687.15800000000002},"last":15.949},{"interface":"front","average":{"1min":26.204000000000001,"5min":26.994,"15min":21.989999999999998},"min":{"1min":11.539999999999999,"5min":8.8379999999999992,"15min":8.1620000000000008},"max":{"1min":108.23,"5min":318.77199999999999,"15min":318.77199999999999},"last":17.994}]},{"osd":5,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":19.896000000000001,"5min":30.462,"15min":24.710999999999999},"min":{"1min":7.9740000000000002,"5min":7.6660000000000004,"15min":6.0650000000000004},"max":{"1min":50.735999999999997,"5min":348.47000000000003,"15min":348.47000000000003},"last":11.254},{"interface":"front","average":{"1min":21.834,"5min":30.434999999999999,"15min":25.710000000000001},"min":{"1min":10.997999999999999,"5min":10.967000000000001,"15min":8.5210000000000008},"max":{"1min":52.564999999999998,"5min":264.41199999999998,"15min":264.41199999999998},"last":16.571000000000002}]},{"osd":6,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":21.768999999999998,"5min":32.344000000000001,"15min":25.484000000000002},"min":{"1min":13.74,"5min":8.5960000000000001,"15min":7.2800000000000002},"max":{"1min":34.429000000000002,"5min":679.51900000000001,"15min":679.51900000000001},"last":19.024999999999999},{"interface":"front","average":{"1min":24.733000000000001,"5min":35.838000000000001,"15min":26.829999999999998},"min":{"1min":15.788,"5min":11.081,"15min":10.121},"max":{"1min":37.802,"5min":676.11800000000005,"15min":676.11800000000005},"last":21.123999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":24.338999999999999,"5min":29.204999999999998,"15min":25.914000000000001},"min":{"1min":12.472,"5min":10.278,"15min":10.278},"max":{"1min":53.186999999999998,"5min":261.48899999999998,"15min":261.48899999999998},"last":20.018999999999998},{"interface":"front","average":{"1min":24.440000000000001,"5min":38.350999999999999,"15min":27.709},"min":{"1min":15.018000000000001,"5min":12.605,"15min":8.4239999999999995},"max":{"1min":42.475000000000001,"5min":678.09500000000003,"15min":678.09500000000003},"last":22.332000000000001}]}]},{"osd":1,"up_from":21,"seq":90194313657,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94792,"kb_used_data":5368,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94277048,"statfs":{"total":96636764160,"available":96539697152,"internally_reserved":0,"allocated":5496832,"data_stored":2730438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":11.568,"5min":16.558,"15min":21.972999999999999},"min":{"1min":4.5970000000000004,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":26.126000000000001,"5min":154.52799999999999,"15min":528.255},"last":15.211},{"interface":"front","average":{"1min":12.198,"5min":15.057,"15min":19.050999999999998},"min":{"1min":4.9619999999999997,"5min":4.5199999999999996,"15min":4.3079999999999998},"max":{"1min":24.762,"5min":127.73999999999999,"15min":447.83199999999999},"last":14.769}]},{"osd":2,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":13.651,"5min":17.736999999999998,"15min":23.350999999999999},"min":{"1min":8.5549999999999997,"5min":7.8719999999999999,"15min":6.194},"max":{"1min":21.834,"5min":116.48999999999999,"15min":420.87700000000001},"last":9.032},{"interface":"front","average":{"1min":15.859999999999999,"5min":26.047999999999998,"15min":27.696999999999999},"min":{"1min":8.5239999999999991,"5min":7.9889999999999999,"15min":7.3209999999999997},"max":{"1min":30.548999999999999,"5min":265.24200000000002,"15min":451.637},"last":18.321000000000002}]},{"osd":3,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":20.800000000000001,"5min":22.111000000000001,"15min":23.620999999999999},"min":{"1min":9.2140000000000004,"5min":8.0370000000000008,"15min":8.0370000000000008},"max":{"1min":62.801000000000002,"5min":141.52099999999999,"15min":419.892},"last":9.6929999999999996},{"interface":"front","average":{"1min":28.942,"5min":25.436,"15min":27.707000000000001},"min":{"1min":8.9109999999999996,"5min":8.6479999999999997,"15min":8.3059999999999992},"max":{"1min":123.798,"5min":143.58699999999999,"15min":425.76100000000002},"last":16.481000000000002}]},{"osd":4,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":26.920999999999999,"5min":26.507999999999999,"15min":25.122},"min":{"1min":11.009,"5min":7.4210000000000003,"15min":7.4210000000000003},"max":{"1min":106.116,"5min":229.63200000000001,"15min":344.48200000000003},"last":12.301},{"interface":"front","average":{"1min":25.748999999999999,"5min":29.904,"15min":30.959},"min":{"1min":12.282,"5min":8.3059999999999992,"15min":8.1259999999999994},"max":{"1min":65.070999999999998,"5min":236.571,"15min":385.43000000000001},"last":17.463000000000001}]},{"osd":5,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":22.766999999999999,"5min":28.076000000000001,"15min":30.946000000000002},"min":{"1min":13.015000000000001,"5min":9.5839999999999996,"15min":9.5839999999999996},"max":{"1min":38.545999999999999,"5min":138.84899999999999,"15min":338.88200000000001},"last":19.140999999999998},{"interface":"front","average":{"1min":18.858000000000001,"5min":26.266999999999999,"15min":29.963999999999999},"min":{"1min":7.577,"5min":7.577,"15min":7.577},"max":{"1min":34.542999999999999,"5min":187.03,"15min":356.423},"last":11.693}]},{"osd":6,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":21.745000000000001,"5min":31.524000000000001,"15min":31.725000000000001},"min":{"1min":10.369999999999999,"5min":10.369999999999999,"15min":9.7289999999999992},"max":{"1min":53.404000000000003,"5min":280.85000000000002,"15min":409.08100000000002},"last":10.302},{"interface":"front","average":{"1min":20.478000000000002,"5min":25.875,"15min":23.920000000000002},"min":{"1min":6.6100000000000003,"5min":6.6100000000000003,"15min":6.6100000000000003},"max":{"1min":59.128999999999998,"5min":243.083,"15min":261.75099999999998},"last":7.8140000000000001}]},{"osd":7,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":21.91,"5min":28.564,"15min":25.913},"min":{"1min":12.441000000000001,"5min":11.923,"15min":7.21},"max":{"1min":43.122999999999998,"5min":169.571,"15min":357.57299999999998},"last":11.053000000000001},{"interface":"front","average":{"1min":23.428000000000001,"5min":23.902999999999999,"15min":25.715},"min":{"1min":8.6929999999999996,"5min":8.1959999999999997,"15min":7.9729999999999999},"max":{"1min":63.999000000000002,"5min":176.809,"15min":418.70600000000002},"last":14.111000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313661,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72248,"kb_used_data":4976,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299592,"statfs":{"total":96636764160,"available":96562782208,"internally_reserved":0,"allocated":5095424,"data_stored":2332598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":12.715,"5min":25.683,"15min":21.652000000000001},"min":{"1min":6.1859999999999999,"5min":6.0220000000000002,"15min":5.7869999999999999},"max":{"1min":33.234999999999999,"5min":246.35400000000001,"15min":301.80599999999998},"last":8.3460000000000001},{"interface":"front","average":{"1min":17.094000000000001,"5min":28.422000000000001,"15min":24.061},"min":{"1min":8.625,"5min":6.0270000000000001,"15min":6.0270000000000001},"max":{"1min":33.862000000000002,"5min":232.851,"15min":242.80000000000001},"last":10.76}]},{"osd":2,"last update":"Sun Jan 21 03:55:44 2024","interfaces":[{"interface":"back","average":{"1min":16.657,"5min":32.222999999999999,"15min":27.050999999999998},"min":{"1min":9.2360000000000007,"5min":7.2160000000000002,"15min":6.6289999999999996},"max":{"1min":28.940000000000001,"5min":279.71300000000002,"15min":312.15800000000002},"last":18.024000000000001},{"interface":"front","average":{"1min":12.459,"5min":35.030999999999999,"15min":27.09},"min":{"1min":6.9169999999999998,"5min":6.3959999999999999,"15min":5.0910000000000002},"max":{"1min":23.015000000000001,"5min":332.57499999999999,"15min":357.38900000000001},"last":11.93}]},{"osd":3,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":43.631,"15min":30.786999999999999},"min":{"1min":11.736000000000001,"5min":7.5049999999999999,"15min":6.9829999999999997},"max":{"1min":56.691000000000003,"5min":547.15800000000002,"15min":547.15800000000002},"last":14.558999999999999},{"interface":"front","average":{"1min":22.646000000000001,"5min":42.786000000000001,"15min":31.776},"min":{"1min":7.4199999999999999,"5min":7.4199999999999999,"15min":7.4199999999999999},"max":{"1min":98.448999999999998,"5min":361.11500000000001,"15min":361.11500000000001},"last":12.789}]},{"osd":4,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":18.094000000000001,"5min":35.396999999999998,"15min":25.135999999999999},"min":{"1min":10.619999999999999,"5min":7.4630000000000001,"15min":7.3019999999999996},"max":{"1min":30.510000000000002,"5min":333.23899999999998,"15min":333.23899999999998},"last":13.444000000000001},{"interface":"front","average":{"1min":27.469000000000001,"5min":42.499000000000002,"15min":29.593},"min":{"1min":7.0999999999999996,"5min":7.0999999999999996,"15min":7.0999999999999996},"max":{"1min":117.73699999999999,"5min":333.87400000000002,"15min":333.87400000000002},"last":15.912000000000001}]},{"osd":5,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":11.044,"5min":37.603999999999999,"15min":30.728000000000002},"min":{"1min":6.617,"5min":6.4249999999999998,"15min":6.4249999999999998},"max":{"1min":25.687999999999999,"5min":387.58100000000002,"15min":543.84100000000001},"last":6.9509999999999996},{"interface":"front","average":{"1min":23.571999999999999,"5min":32.238999999999997,"15min":23.466999999999999},"min":{"1min":11.239000000000001,"5min":6.3529999999999998,"15min":5.452},"max":{"1min":42.279000000000003,"5min":266.03800000000001,"15min":335.589},"last":15.298}]},{"osd":6,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":16.440000000000001,"5min":32.234000000000002,"15min":25.504999999999999},"min":{"1min":6.7729999999999997,"5min":6.1059999999999999,"15min":6.1059999999999999},"max":{"1min":30.492000000000001,"5min":311.358,"15min":370.30200000000002},"last":9.6910000000000007},{"interface":"front","average":{"1min":19.751000000000001,"5min":33.356000000000002,"15min":26.114999999999998},"min":{"1min":11.441000000000001,"5min":8.2769999999999992,"15min":7.2389999999999999},"max":{"1min":40.555,"5min":230.27699999999999,"15min":230.27699999999999},"last":19.289999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":20.992000000000001,"5min":30.393999999999998,"15min":25.266999999999999},"min":{"1min":11.186,"5min":9.1110000000000007,"15min":8.8960000000000008},"max":{"1min":44.409999999999997,"5min":131.65000000000001,"15min":214.071},"last":18.661999999999999},{"interface":"front","average":{"1min":23.757999999999999,"5min":38.738999999999997,"15min":28.484999999999999},"min":{"1min":9.0990000000000002,"5min":8.8070000000000004,"15min":8.8070000000000004},"max":{"1min":53.152000000000001,"5min":348.38600000000002,"15min":348.38600000000002},"last":17.190999999999999}]}]},{"osd":6,"up_from":18,"seq":77309411772,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72256,"kb_used_data":4980,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299584,"statfs":{"total":96636764160,"available":96562774016,"internally_reserved":0,"allocated":5099520,"data_stored":2332603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":16.803000000000001,"5min":22.076000000000001,"15min":21.289999999999999},"min":{"1min":5.3380000000000001,"5min":5.3380000000000001,"15min":4.391},"max":{"1min":98.180000000000007,"5min":284.42500000000001,"15min":438.98899999999998},"last":9.7970000000000006},{"interface":"front","average":{"1min":28.359999999999999,"5min":27.478999999999999,"15min":21.431000000000001},"min":{"1min":8.7149999999999999,"5min":5.2160000000000002,"15min":4.8840000000000003},"max":{"1min":144.92400000000001,"5min":380.07600000000002,"15min":380.07600000000002},"last":14.294}]},{"osd":1,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":19.898,"5min":31.847999999999999,"15min":24.283000000000001},"min":{"1min":8.7289999999999992,"5min":6.883,"15min":4.7389999999999999},"max":{"1min":54.722999999999999,"5min":526.48099999999999,"15min":526.48099999999999},"last":11.242000000000001},{"interface":"front","average":{"1min":14.859999999999999,"5min":26.492000000000001,"15min":22.867999999999999},"min":{"1min":6.3600000000000003,"5min":4.3120000000000003,"15min":4.3120000000000003},"max":{"1min":44.970999999999997,"5min":395.72399999999999,"15min":395.72399999999999},"last":6.1980000000000004}]},{"osd":2,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":12.648,"5min":21.052,"15min":20.300999999999998},"min":{"1min":6.0350000000000001,"5min":5.8920000000000003,"15min":5.8920000000000003},"max":{"1min":35.268999999999998,"5min":209.78,"15min":213.83199999999999},"last":7.0060000000000002},{"interface":"front","average":{"1min":22.007000000000001,"5min":31.407,"15min":26.654},"min":{"1min":6.7210000000000001,"5min":6.7210000000000001,"15min":6.7210000000000001},"max":{"1min":77.159000000000006,"5min":444.822,"15min":444.822},"last":15.56}]},{"osd":3,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":18.542999999999999,"5min":33.561,"15min":26.021000000000001},"min":{"1min":8.1959999999999997,"5min":6.1890000000000001,"15min":6.1890000000000001},"max":{"1min":59.645000000000003,"5min":527.40899999999999,"15min":527.40899999999999},"last":10.084},{"interface":"front","average":{"1min":17.805,"5min":25.48,"15min":23.213999999999999},"min":{"1min":6.4089999999999998,"5min":6.4050000000000002,"15min":6.4050000000000002},"max":{"1min":57.729999999999997,"5min":370.10000000000002,"15min":370.10000000000002},"last":11.903}]},{"osd":4,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":21.780000000000001,"5min":31.358000000000001,"15min":24.521999999999998},"min":{"1min":6.9160000000000004,"5min":6.3120000000000003,"15min":6.3120000000000003},"max":{"1min":95.962000000000003,"5min":385.49200000000002,"15min":385.49200000000002},"last":17.129000000000001},{"interface":"front","average":{"1min":21.581,"5min":26.248000000000001,"15min":21.954000000000001},"min":{"1min":8.0660000000000007,"5min":7.2130000000000001,"15min":6.7539999999999996},"max":{"1min":89.241,"5min":241.59200000000001,"15min":241.59200000000001},"last":12.952}]},{"osd":5,"last update":"Sun Jan 21 03:56:31 2024","interfaces":[{"interface":"back","average":{"1min":22.273,"5min":28.437000000000001,"15min":25.361999999999998},"min":{"1min":8.9830000000000005,"5min":8.2810000000000006,"15min":7.9450000000000003},"max":{"1min":98.718000000000004,"5min":488.01299999999998,"15min":488.01299999999998},"last":17.882999999999999},{"interface":"front","average":{"1min":21.077000000000002,"5min":23.109000000000002,"15min":25.189},"min":{"1min":9.093,"5min":8.1890000000000001,"15min":8.1890000000000001},"max":{"1min":101.967,"5min":197.38,"15min":333.43900000000002},"last":9.093}]},{"osd":7,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":23.504000000000001,"5min":30.591999999999999,"15min":25.545000000000002},"min":{"1min":10.676,"5min":7.1340000000000003,"15min":7.1340000000000003},"max":{"1min":100.84399999999999,"5min":322.54599999999999,"15min":322.54599999999999},"last":14.961},{"interface":"front","average":{"1min":24.988,"5min":29.010999999999999,"15min":25.050000000000001},"min":{"1min":13.025,"5min":9.6940000000000008,"15min":8.6699999999999999},"max":{"1min":90.594999999999999,"5min":218.30199999999999,"15min":218.30199999999999},"last":13.550000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411775,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72640,"kb_used_data":5368,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299200,"statfs":{"total":96636764160,"available":96562380800,"internally_reserved":0,"allocated":5496832,"data_stored":2730438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":9.1660000000000004,"5min":22.483000000000001,"15min":20.535},"min":{"1min":4.9420000000000002,"5min":4.6539999999999999,"15min":4.0940000000000003},"max":{"1min":28.151,"5min":241.49000000000001,"15min":346.541},"last":9.8379999999999992},{"interface":"front","average":{"1min":13.221,"5min":26.934000000000001,"15min":26.605},"min":{"1min":7.6550000000000002,"5min":6.1500000000000004,"15min":5.8819999999999997},"max":{"1min":24.053000000000001,"5min":236.81100000000001,"15min":524.904},"last":16.914000000000001}]},{"osd":1,"last update":"Sun Jan 21 03:56:24 2024","interfaces":[{"interface":"back","average":{"1min":12.57,"5min":24.992999999999999,"15min":19.875},"min":{"1min":7.0039999999999996,"5min":5.1429999999999998,"15min":4.907},"max":{"1min":22.702999999999999,"5min":284.07400000000001,"15min":368.12400000000002},"last":7.6580000000000004},{"interface":"front","average":{"1min":19.931000000000001,"5min":33.564999999999998,"15min":26.795000000000002},"min":{"1min":8.7680000000000007,"5min":8.7680000000000007,"15min":7.5359999999999996},"max":{"1min":42.582999999999998,"5min":296.56400000000002,"15min":368.86099999999999},"last":16.274999999999999}]},{"osd":2,"last update":"Sun Jan 21 03:55:35 2024","interfaces":[{"interface":"back","average":{"1min":11.273999999999999,"5min":46.090000000000003,"15min":29.077000000000002},"min":{"1min":7.25,"5min":6.6310000000000002,"15min":6.0579999999999998},"max":{"1min":27.489000000000001,"5min":2166.2739999999999,"15min":2166.2739999999999},"last":10.122},{"interface":"front","average":{"1min":19.923999999999999,"5min":53.329000000000001,"15min":37.494},"min":{"1min":9.8030000000000008,"5min":7.1150000000000002,"15min":7.1150000000000002},"max":{"1min":71.177999999999997,"5min":2181.6190000000001,"15min":2181.6190000000001},"last":18.687000000000001}]},{"osd":3,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":16.638000000000002,"5min":30.282,"15min":26.837},"min":{"1min":6.8179999999999996,"5min":6.2770000000000001,"15min":6.2770000000000001},"max":{"1min":63.612000000000002,"5min":412.714,"15min":412.714},"last":12.858000000000001},{"interface":"front","average":{"1min":20.707999999999998,"5min":34.683999999999997,"15min":28.085999999999999},"min":{"1min":9.9399999999999995,"5min":9.6389999999999993,"15min":6.8840000000000003},"max":{"1min":57.048000000000002,"5min":314.28199999999998,"15min":314.28199999999998},"last":13.99}]},{"osd":4,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":21.478000000000002,"5min":39.764000000000003,"15min":27.739999999999998},"min":{"1min":12.226000000000001,"5min":7.7060000000000004,"15min":6.274},"max":{"1min":49.774999999999999,"5min":418.18299999999999,"15min":418.18299999999999},"last":9.4589999999999996},{"interface":"front","average":{"1min":24.219000000000001,"5min":42.651000000000003,"15min":30.201000000000001},"min":{"1min":14.734999999999999,"5min":11.993,"15min":7.5490000000000004},"max":{"1min":57.712000000000003,"5min":387.125,"15min":387.125},"last":14.619999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":17.492999999999999,"5min":32.762,"15min":25.588999999999999},"min":{"1min":8.6739999999999995,"5min":6.7050000000000001,"15min":6.7050000000000001},"max":{"1min":31.645,"5min":431.40899999999999,"15min":431.40899999999999},"last":13.457000000000001},{"interface":"front","average":{"1min":23.259,"5min":36.051000000000002,"15min":28.965},"min":{"1min":11.241,"5min":9.0809999999999995,"15min":8.5470000000000006},"max":{"1min":61.225999999999999,"5min":432.19,"15min":432.19},"last":17.981999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":23.452000000000002,"5min":38.652000000000001,"15min":29.713000000000001},"min":{"1min":13.459,"5min":9.7560000000000002,"15min":7.8639999999999999},"max":{"1min":48.789999999999999,"5min":281.06099999999998,"15min":281.06099999999998},"last":11.839},{"interface":"front","average":{"1min":12.581,"5min":26.823,"15min":22.974},"min":{"1min":6.5170000000000003,"5min":6.5170000000000003,"15min":6.5170000000000003},"max":{"1min":36.563000000000002,"5min":282.29000000000002,"15min":282.29000000000002},"last":12.227}]}]},{"osd":7,"up_from":14,"seq":60129542588,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72264,"kb_used_data":4980,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299576,"statfs":{"total":96636764160,"available":96562765824,"internally_reserved":0,"allocated":5099520,"data_stored":2332603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:17 2024","interfaces":[{"interface":"back","average":{"1min":11.395,"5min":20.713000000000001,"15min":16.225999999999999},"min":{"1min":5.0099999999999998,"5min":4.7450000000000001,"15min":4.1929999999999996},"max":{"1min":27.079000000000001,"5min":239.37799999999999,"15min":304.20800000000003},"last":8.0229999999999997},{"interface":"front","average":{"1min":10.247999999999999,"5min":21.539000000000001,"15min":16.581},"min":{"1min":5.9039999999999999,"5min":5.2709999999999999,"15min":4.2679999999999998},"max":{"1min":31.059999999999999,"5min":181.04599999999999,"15min":181.04599999999999},"last":12.866}]},{"osd":1,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":12.342000000000001,"5min":19.533000000000001,"15min":20.614000000000001},"min":{"1min":6.0869999999999997,"5min":5.6580000000000004,"15min":5.0419999999999998},"max":{"1min":27.402999999999999,"5min":138.82499999999999,"15min":248.17500000000001},"last":10.874000000000001},{"interface":"front","average":{"1min":14.029,"5min":23.824999999999999,"15min":23.670999999999999},"min":{"1min":7.9109999999999996,"5min":6.3769999999999998,"15min":6.3769999999999998},"max":{"1min":26.454000000000001,"5min":214.55000000000001,"15min":277.85700000000003},"last":10.332000000000001}]},{"osd":2,"last update":"Sun Jan 21 03:56:17 2024","interfaces":[{"interface":"back","average":{"1min":17.594999999999999,"5min":36.784999999999997,"15min":27.683},"min":{"1min":7.2869999999999999,"5min":7.2290000000000001,"15min":6.1779999999999999},"max":{"1min":47.972999999999999,"5min":683.21000000000004,"15min":683.21000000000004},"last":9.75},{"interface":"front","average":{"1min":16.43,"5min":24.434999999999999,"15min":22.135999999999999},"min":{"1min":9.0399999999999991,"5min":6.3890000000000002,"15min":6.3890000000000002},"max":{"1min":35.869,"5min":150.369,"15min":367.30000000000001},"last":17.41}]},{"osd":3,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":17.163,"5min":27.100000000000001,"15min":22.632000000000001},"min":{"1min":10.734999999999999,"5min":7.1470000000000002,"15min":6.8040000000000003},"max":{"1min":24.895,"5min":192.14599999999999,"15min":192.14599999999999},"last":14.135},{"interface":"front","average":{"1min":16.158000000000001,"5min":24.597000000000001,"15min":21.649000000000001},"min":{"1min":7.851,"5min":7.8090000000000002,"15min":6.5960000000000001},"max":{"1min":29.548999999999999,"5min":166.846,"15min":210.006},"last":8.8729999999999993}]},{"osd":4,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":20.638999999999999,"5min":27.516999999999999,"15min":22.658999999999999},"min":{"1min":9.4700000000000006,"5min":7.6260000000000003,"15min":7.2309999999999999},"max":{"1min":43.503,"5min":197.31,"15min":268.416},"last":11.632},{"interface":"front","average":{"1min":17.760999999999999,"5min":25.548999999999999,"15min":21.73},"min":{"1min":7.516,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":33.429000000000002,"5min":143.07400000000001,"15min":168.95599999999999},"last":14.869}]},{"osd":5,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":17.228999999999999,"5min":25.649999999999999,"15min":24.148},"min":{"1min":7.3029999999999999,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":25.670000000000002,"5min":149.56,"15min":226.947},"last":13.496},{"interface":"front","average":{"1min":17.081,"5min":24.050000000000001,"15min":23.798999999999999},"min":{"1min":8.6419999999999995,"5min":7.5839999999999996,"15min":7.5839999999999996},"max":{"1min":41.097000000000001,"5min":137.00200000000001,"15min":248.94800000000001},"last":9.4529999999999994}]},{"osd":6,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":21.542000000000002,"5min":27.488,"15min":26.065999999999999},"min":{"1min":10.897,"5min":9.4459999999999997,"15min":9.4459999999999997},"max":{"1min":55.081000000000003,"5min":193.16,"15min":295.12099999999998},"last":12.221},{"interface":"front","average":{"1min":19.73,"5min":28.16,"15min":26.513999999999999},"min":{"1min":11.029,"5min":9.3219999999999992,"15min":9.3219999999999992},"max":{"1min":55.758000000000003,"5min":178.01900000000001,"15min":278.71899999999999},"last":18.251000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542587,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72256,"kb_used_data":4976,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299584,"statfs":{"total":96636764160,"available":96562774016,"internally_reserved":0,"allocated":5095424,"data_stored":2332598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":8.7889999999999997,"5min":15.209,"15min":16.693000000000001},"min":{"1min":4.9939999999999998,"5min":4.4660000000000002,"15min":4.4100000000000001},"max":{"1min":18.459,"5min":220.72900000000001,"15min":220.72900000000001},"last":7.8449999999999998},{"interface":"front","average":{"1min":12.965,"5min":16.300000000000001,"15min":18.149999999999999},"min":{"1min":6.6360000000000001,"5min":4.6029999999999998,"15min":4.6029999999999998},"max":{"1min":45.616,"5min":101.768,"15min":204.142},"last":13.081}]},{"osd":1,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":12.475,"5min":22.411999999999999,"15min":18.585000000000001},"min":{"1min":6.6440000000000001,"5min":4.9359999999999999,"15min":4.9359999999999999},"max":{"1min":26.292999999999999,"5min":272.61399999999998,"15min":272.61399999999998},"last":8.827},{"interface":"front","average":{"1min":16.969000000000001,"5min":24.193000000000001,"15min":20.829000000000001},"min":{"1min":9.1150000000000002,"5min":6.8719999999999999,"15min":5.6200000000000001},"max":{"1min":49.414999999999999,"5min":142.97300000000001,"15min":223.49299999999999},"last":13.942}]},{"osd":2,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":23.027000000000001,"5min":29.632000000000001,"15min":24.187999999999999},"min":{"1min":9.3979999999999997,"5min":7.4320000000000004,"15min":6.6879999999999997},"max":{"1min":77.858999999999995,"5min":285.61000000000001,"15min":285.61000000000001},"last":9.1069999999999993},{"interface":"front","average":{"1min":21.07,"5min":26.555,"15min":22.710999999999999},"min":{"1min":7.6769999999999996,"5min":6.899,"15min":6.2080000000000002},"max":{"1min":101.676,"5min":298.892,"15min":298.892},"last":15.271000000000001}]},{"osd":3,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":14.119,"5min":19.032,"15min":20.623999999999999},"min":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"max":{"1min":48.497,"5min":132.958,"15min":200.697},"last":9.9540000000000006},{"interface":"front","average":{"1min":19.265000000000001,"5min":23.931999999999999,"15min":21.149000000000001},"min":{"1min":8.9890000000000008,"5min":7.5250000000000004,"15min":7.1539999999999999},"max":{"1min":60.786000000000001,"5min":166.66200000000001,"15min":188.06399999999999},"last":16.486000000000001}]},{"osd":5,"last update":"Sun Jan 21 03:55:58 2024","interfaces":[{"interface":"back","average":{"1min":17.315000000000001,"5min":26.736000000000001,"15min":24.969000000000001},"min":{"1min":10.885999999999999,"5min":8.2970000000000006,"15min":6.7359999999999998},"max":{"1min":30.718,"5min":182.53700000000001,"15min":470.87700000000001},"last":15.875999999999999},{"interface":"front","average":{"1min":13.831,"5min":23.012,"15min":24.087},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":6.944},"max":{"1min":23.405000000000001,"5min":145.125,"15min":471.56999999999999},"last":11.353999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:55:37 2024","interfaces":[{"interface":"back","average":{"1min":21.370999999999999,"5min":25.059999999999999,"15min":21.741},"min":{"1min":14.724,"5min":8.5820000000000007,"15min":6.415},"max":{"1min":73.459999999999994,"5min":208.42099999999999,"15min":285.87},"last":11.964},{"interface":"front","average":{"1min":18.635000000000002,"5min":26.960000000000001,"15min":24.614000000000001},"min":{"1min":9.5129999999999999,"5min":9.5129999999999999,"15min":7.5430000000000001},"max":{"1min":64.510000000000005,"5min":252.47999999999999,"15min":252.47999999999999},"last":14.577}]},{"osd":7,"last update":"Sun Jan 21 03:55:58 2024","interfaces":[{"interface":"back","average":{"1min":20.036999999999999,"5min":25.931000000000001,"15min":22.280000000000001},"min":{"1min":12.244,"5min":9.3040000000000003,"15min":7.1820000000000004},"max":{"1min":44.164999999999999,"5min":141.51300000000001,"15min":229.04900000000001},"last":18.936},{"interface":"front","average":{"1min":22.382999999999999,"5min":28.206,"15min":22.698},"min":{"1min":15.759,"5min":12.417999999999999,"15min":7.4299999999999997},"max":{"1min":42.031999999999996,"5min":179.12799999999999,"15min":179.12799999999999},"last":18.146000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T03:56:36.475 INFO:tasks.ceph:pgid 14.1e last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.475 INFO:tasks.ceph:pgid 14.1f last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.475 INFO:tasks.ceph:pgid 14.1c last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.475 INFO:tasks.ceph:pgid 14.1d last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.475 INFO:tasks.ceph:pgid 14.2 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.475 INFO:tasks.ceph:pgid 14.3 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.475 INFO:tasks.ceph:pgid 14.0 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.475 INFO:tasks.ceph:pgid 14.1 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.476 INFO:tasks.ceph:pgid 14.7 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.476 INFO:tasks.ceph:pgid 14.4 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.476 INFO:tasks.ceph:pgid 14.5 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.476 INFO:tasks.ceph:pgid 14.f last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.476 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-21T03:19:59.904069+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=19, tm_sec=59, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.476 INFO:tasks.ceph:pgid 14.c last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.476 INFO:tasks.ceph:pgid 14.d last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.476 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-01-21T03:19:59.904069+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=19, tm_sec=59, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.476 INFO:tasks.ceph:pgid 14.e last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.477 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-21T03:19:59.904069+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=19, tm_sec=59, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.477 INFO:tasks.ceph:pgid 14.8 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.477 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-01-21T03:19:59.904069+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=19, tm_sec=59, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.477 INFO:tasks.ceph:pgid 14.a last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.477 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-21T03:19:59.904069+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=19, tm_sec=59, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.477 INFO:tasks.ceph:pgid 14.b last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.477 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-21T03:19:59.904069+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=19, tm_sec=59, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.477 INFO:tasks.ceph:pgid 14.19 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.477 INFO:tasks.ceph:pgid 14.18 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.477 INFO:tasks.ceph:pgid 14.1b last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.478 INFO:tasks.ceph:pgid 14.1a last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.478 INFO:tasks.ceph:pgid 14.15 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.478 INFO:tasks.ceph:pgid 14.14 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.478 INFO:tasks.ceph:pgid 14.17 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.478 INFO:tasks.ceph:pgid 14.16 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.478 INFO:tasks.ceph:pgid 14.11 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.478 INFO:tasks.ceph:pgid 14.10 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.478 INFO:tasks.ceph:pgid 14.13 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.478 INFO:tasks.ceph:pgid 14.12 last_scrub_stamp 2024-01-21T03:21:02.392695+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=21, tm_sec=2, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=3, tm_min=56, tm_sec=28, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T03:56:36.478 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-21T03:56:36.479 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:50.793 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:36.484 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:50.798 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:36.500 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:37:50.815 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:51.465 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:05.778 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:51.467 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:05.782 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:51.488 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:05.803 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:51.519 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:05.778 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:51.520 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:05.780 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:51.520 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:05.835 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:51.524 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:05.838 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:51.540 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:05.799 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:51.542 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:05.856 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:52.273 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:06.532 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:52.274 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:06.533 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:52.282 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:06.541 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:52.328 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:06.642 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:52.329 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:06.643 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:52.338 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:06.652 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:52.368 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:06.682 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:52.369 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:06.683 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:52.377 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:06.692 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:56:56.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T03:56:56.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.585+0000 7f1b12d5e640 10 monclient: get_monmap_and_config 2024-01-21T03:56:56.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.585+0000 7f1b12d5e640 10 monclient: build_initial_monmap 2024-01-21T03:56:56.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.585+0000 7f1b12d5e640 10 monclient: monmap: 2024-01-21T03:56:56.586 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:56.586 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:56.586 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:56.586593+0000 2024-01-21T03:56:56.586 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:56.586593+0000 2024-01-21T03:56:56.586 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:56.587 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.586+0000 7f1b12d5e640 1 Processor -- start 2024-01-21T03:56:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.586+0000 7f1b12d5e640 1 -- start start 2024-01-21T03:56:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.586+0000 7f1b12d5e640 10 monclient: init 2024-01-21T03:56:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.586+0000 7f1b12d5e640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.586+0000 7f1b12d5e640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-01-21T03:56:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.586+0000 7f1b12d5e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1b0c0b12d0 0x7f1b0c141f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.586+0000 7f1b12d5e640 10 monclient(hunting): picked mon.noname-a con 0x7f1b0c0b12d0 addr v2:172.21.15.7:3300/0 2024-01-21T03:56:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.586+0000 7f1b12d5e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1b0c142b90 0x7f1b0c0b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.586+0000 7f1b12d5e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b0c142b90 addr v2:172.21.15.177:3300/0 2024-01-21T03:56:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.586+0000 7f1b12d5e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b0c06c980 addr v1:172.21.15.7:6789/0 2024-01-21T03:56:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.586+0000 7f1b12d5e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0c142670 con 0x7f1b0c06c980 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.586+0000 7f1b12d5e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.586+0000 7f1b12d5e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b0c0b1bf0 con 0x7f1b0c0b12d0 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.587+0000 7f1b12d5e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.587+0000 7f1b12d5e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b0c070380 con 0x7f1b0c142b90 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.587+0000 7f1b12d5e640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.587+0000 7f1b12d5e640 10 monclient(hunting): authenticate will time out at 3562.902758s 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.587+0000 7f1b10afa640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1b0c0b12d0 0x7f1b0c141f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.587+0000 7f1b03fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1b0c142b90 0x7f1b0c0b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.587+0000 7f1b03fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1b0c142b90 0x7f1b0c0b16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:34124/0 (socket says 172.21.15.7:34124) 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.587+0000 7f1b03fff640 1 -- 172.21.15.7:0/405195336 learned_addr learned my addr 172.21.15.7:0/405195336 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.587+0000 7f1b03fff640 10 monclient(hunting): get_auth_request con 0x7f1b0c142b90 auth_method 0 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.587+0000 7f1b03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.587+0000 7f1b03fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.587+0000 7f1b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.588+0000 7f1b10afa640 10 monclient(hunting): get_auth_request con 0x7f1b0c0b12d0 auth_method 0 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.588+0000 7f1b10afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.588+0000 7f1b10afa640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:56.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.588+0000 7f1b10afa640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:56.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.588+0000 7f1b03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:56.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.588+0000 7f1b03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:56.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.588+0000 7f1b03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:56.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.589+0000 7f1b10afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:56.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.589+0000 7f1b10afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.589+0000 7f1b10afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.589+0000 7f1b03fff640 10 monclient(hunting): handle_auth_done global_id 23245 payload 274 2024-01-21T03:56:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.589+0000 7f1b03fff640 1 -- 172.21.15.7:0/405195336 >> v2:172.21.15.7:3300/0 conn(0x7f1b0c0b12d0 msgr2=0x7f1b0c141f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T03:56:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.589+0000 7f1b03fff640 1 --2- 172.21.15.7:0/405195336 >> v2:172.21.15.7:3300/0 conn(0x7f1b0c0b12d0 0x7f1b0c141f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.590+0000 7f1b03fff640 1 -- 172.21.15.7:0/405195336 >> v1:172.21.15.7:6789/0 conn(0x7f1b0c06c980 legacy=0x7f1b0c145450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.590+0000 7f1b03fff640 10 monclient: _finish_hunting 0 2024-01-21T03:56:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.590+0000 7f1b03fff640 1 monclient: found mon.noname-c 2024-01-21T03:56:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.590+0000 7f1b03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.590+0000 7f1b03fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-01-21T03:56:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.590+0000 7f1b03fff640 1 -- 172.21.15.7:0/405195336 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0c0704f0 con 0x7f1b0c142b90 2024-01-21T03:56:56.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.591+0000 7f1b03fff640 1 --2- 172.21.15.7:0/405195336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b0c142b90 0x7f1b0c0b16b0 secure :-1 s=READY pgs=5833 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0b0190 tx=0x7f1afc02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd06ce903faec86b server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b037fe640 1 -- 172.21.15.7:0/405195336 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc031870 con 0x7f1b0c142b90 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b037fe640 10 monclient: dump: 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b037fe640 10 monclient: _finish_auth 0 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b037fe640 10 monclient: _check_auth_tickets 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b037fe640 1 -- 172.21.15.7:0/405195336 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1afc0319d0 con 0x7f1b0c142b90 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b037fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b037fe640 1 -- 172.21.15.7:0/405195336 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc031d60 con 0x7f1b0c142b90 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b037fe640 10 monclient: dump: 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b12d5e640 5 monclient: authenticate success, global_id 23245 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b12d5e640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.595+0000 7f1b12d5e640 4 set_mon_vals no callback set 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.596+0000 7f1b12d5e640 10 monclient: shutdown 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.596+0000 7f1b12d5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.596+0000 7f1b12d5e640 1 -- 172.21.15.7:0/405195336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b0c142b90 msgr2=0x7f1b0c0b16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.596+0000 7f1b12d5e640 1 --2- 172.21.15.7:0/405195336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b0c142b90 0x7f1b0c0b16b0 secure :-1 s=READY pgs=5833 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0b0190 tx=0x7f1afc02b910 comp rx=0 tx=0).stop 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.596+0000 7f1b12d5e640 1 -- 172.21.15.7:0/405195336 shutdown_connections 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.596+0000 7f1b12d5e640 1 --2- 172.21.15.7:0/405195336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b0c142b90 0x7f1b0c0b16b0 unknown :-1 s=CLOSED pgs=5833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.596+0000 7f1b12d5e640 1 --2- 172.21.15.7:0/405195336 >> v2:172.21.15.7:3300/0 conn(0x7f1b0c0b12d0 0x7f1b0c141f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.596+0000 7f1b12d5e640 1 -- 172.21.15.7:0/405195336 >> 172.21.15.7:0/405195336 conn(0x7f1b0c0b23c0 msgr2=0x7f1b0c0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.596+0000 7f1b12d5e640 1 -- 172.21.15.7:0/405195336 shutdown_connections 2024-01-21T03:56:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.596+0000 7f1b12d5e640 1 -- 172.21.15.7:0/405195336 wait complete. 2024-01-21T03:56:56.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.597+0000 7f1b12d5e640 10 monclient: build_initial_monmap 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.597+0000 7f1b12d5e640 10 monclient: monmap: 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.597+0000 7f1b12d5e640 1 librados: starting msgr at 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.597+0000 7f1b12d5e640 1 librados: starting objecter 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.597+0000 7f1b12d5e640 1 Processor -- start 2024-01-21T03:56:56.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.597+0000 7f1b12d5e640 1 -- start start 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.597+0000 7f1b12d5e640 1 librados: setting wanted keys 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.597+0000 7f1b12d5e640 1 librados: calling monclient init 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.597+0000 7f1b12d5e640 10 monclient: init 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b0c06c980 0x7f1b0c13d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 10 monclient(hunting): picked mon.noname-a con 0x7f1b0c06c980 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b0c0b12d0 0x7f1b0c13adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b0c0b12d0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b0c142b90 0x7f1b0c1da4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 10 monclient(hunting): picked mon.noname-b con 0x7f1b0c142b90 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c13a930 con 0x7f1b0c06c980 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:56.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c0ace20 con 0x7f1b0c142b90 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c0ad100 con 0x7f1b0c0b12d0 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b12d5e640 10 monclient(hunting): authenticate will time out at 3562.913878s 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b10afa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b0c06c980 0x7f1b0c13d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b112fb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b0c142b90 0x7f1b0c1da4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b03fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b0c0b12d0 0x7f1b0c13adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b10afa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b0c06c980 0x7f1b0c13d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52748/0 (socket says 172.21.15.7:52748) 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b10afa640 1 -- 172.21.15.7:0/4078058781 learned_addr learned my addr 172.21.15.7:0/4078058781 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b10afa640 10 monclient(hunting): get_auth_request con 0x7f1b0c06c980 auth_method 0 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b10afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b10afa640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b10afa640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b112fb640 10 monclient(hunting): get_auth_request con 0x7f1b0c142b90 auth_method 0 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.598+0000 7f1b112fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b112fb640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b112fb640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b03fff640 10 monclient(hunting): get_auth_request con 0x7f1b0c0b12d0 auth_method 0 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b03fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b112fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b112fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b112fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b10afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b10afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b10afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.599+0000 7f1b03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.602+0000 7f1b112fb640 10 monclient(hunting): handle_auth_done global_id 23248 payload 824 2024-01-21T03:56:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.602+0000 7f1b112fb640 1 -- 172.21.15.7:0/4078058781 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b0c0b12d0 msgr2=0x7f1b0c13adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:56.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.602+0000 7f1b112fb640 1 --2- 172.21.15.7:0/4078058781 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b0c0b12d0 0x7f1b0c13adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:56.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.602+0000 7f1b112fb640 1 -- 172.21.15.7:0/4078058781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b0c06c980 msgr2=0x7f1b0c13d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:56.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.602+0000 7f1b112fb640 1 --2- 172.21.15.7:0/4078058781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b0c06c980 0x7f1b0c13d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:56.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.602+0000 7f1b112fb640 10 monclient: _finish_hunting 0 2024-01-21T03:56:56.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.602+0000 7f1b112fb640 1 monclient: found mon.noname-b 2024-01-21T03:56:56.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.602+0000 7f1b112fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:56.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.602+0000 7f1b112fb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:56:56.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.602+0000 7f1b112fb640 1 -- 172.21.15.7:0/4078058781 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc0094d0 con 0x7f1b0c142b90 2024-01-21T03:56:56.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.604+0000 7f1b112fb640 1 --2- 172.21.15.7:0/4078058781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b0c142b90 0x7f1b0c1da4d0 secure :-1 s=READY pgs=5834 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400c690 tx=0x7f1b0400cb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:56.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.604+0000 7f1b017fa640 10 client.?.objecter ms_handle_connect 0x7f1b0c142b90 2024-01-21T03:56:56.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.604+0000 7f1b017fa640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:56.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.607+0000 7f1b017fa640 1 -- 172.21.15.7:0/4078058781 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b04003b70 con 0x7f1b0c142b90 2024-01-21T03:56:56.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.607+0000 7f1b017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:56.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.607+0000 7f1b017fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:56.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.607+0000 7f1b017fa640 10 monclient: dump: 2024-01-21T03:56:56.609 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:56.609 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:56.609 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:56.609 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.607+0000 7f1b017fa640 10 monclient: _finish_auth 0 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.607+0000 7f1b017fa640 10 monclient: _check_auth_tickets 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b017fa640 1 -- 172.21.15.7:0/4078058781 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b04003cd0 con 0x7f1b0c142b90 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b12d5e640 5 monclient: authenticate success, global_id 23248 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b017fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b12d5e640 10 monclient: _renew_subs 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b017fa640 1 -- 172.21.15.7:0/4078058781 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b04005310 con 0x7f1b0c142b90 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b12d5e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:56.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b12d5e640 1 -- 172.21.15.7:0/4078058781 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0c13a550 con 0x7f1b0c142b90 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b017fa640 10 monclient: dump: 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b027fc640 4 set_mon_vals no callback set 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b12d5e640 10 client.23248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b12d5e640 10 monclient: _renew_subs 2024-01-21T03:56:56.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b12d5e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:56.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b12d5e640 1 -- 172.21.15.7:0/4078058781 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0c1dabf0 con 0x7f1b0c142b90 2024-01-21T03:56:56.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b12d5e640 1 librados: init done 2024-01-21T03:56:56.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b12d5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:56.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b12d5e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:56.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b12d5e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:56.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.608+0000 7f1b12d5e640 1 -- 172.21.15.7:0/4078058781 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0c13a740 con 0x7f1b0c142b90 2024-01-21T03:56:56.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.628+0000 7f1b017fa640 1 -- 172.21.15.7:0/4078058781 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1b04005470 con 0x7f1b0c142b90 2024-01-21T03:56:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.628+0000 7f1b017fa640 1 --2- 172.21.15.7:0/4078058781 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f1af806c8b0 0x7f1af806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.628+0000 7f1b10afa640 1 --2- 172.21.15.7:0/4078058781 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f1af806c8b0 0x7f1af806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.628+0000 7f1b10afa640 10 monclient: get_auth_request con 0x7f1af806c8b0 auth_method 0 2024-01-21T03:56:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.628+0000 7f1b017fa640 10 client.23248.objecter ms_handle_connect 0x7f1af806c8b0 2024-01-21T03:56:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.628+0000 7f1b10afa640 1 --2- 172.21.15.7:0/4078058781 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f1af806c8b0 0x7f1af806ed70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1b08009bf0 tx=0x7f1b08009110 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:56.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.849+0000 7f1b017fa640 1 -- 172.21.15.7:0/4078058781 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7f1b04058980 con 0x7f1b0c142b90 2024-01-21T03:56:56.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.849+0000 7f1b017fa640 10 client.23248.objecter ms_dispatch 0x7f1b0c000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:56.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.849+0000 7f1b017fa640 3 client.23248.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:56.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.849+0000 7f1b017fa640 3 client.23248.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:56.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.849+0000 7f1b017fa640 20 client.23248.objecter dump_active .. 0 homeless 2024-01-21T03:56:56.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.859+0000 7f1b017fa640 1 -- 172.21.15.7:0/4078058781 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1b04054cc0 con 0x7f1b0c142b90 2024-01-21T03:56:56.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.859+0000 7f1b017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:56.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:56.859+0000 7f1b017fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:57.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.041+0000 7f1b12d5e640 1 -- 172.21.15.7:0/4078058781 --> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1b0c0acf60 con 0x7f1af806c8b0 2024-01-21T03:56:57.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.046+0000 7f1b017fa640 1 -- 172.21.15.7:0/4078058781 <== mgr.4103 v2:172.21.15.7:6800/2590781548 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151954 (secure 0 0 0) 0x7f1b0c0acf60 con 0x7f1af806c8b0 2024-01-21T03:56:57.048 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-01-21T03:56:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-01-21T03:56:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 10 librados: watch_flush enter 2024-01-21T03:56:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 10 librados: watch_flush exit 2024-01-21T03:56:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 20 client.23248.objecter shutdown clearing up homeless session... 2024-01-21T03:56:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 10 client.23248.objecter successfully canceled tick 2024-01-21T03:56:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 1 -- 172.21.15.7:0/4078058781 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f1af806c8b0 msgr2=0x7f1af806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 1 --2- 172.21.15.7:0/4078058781 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f1af806c8b0 0x7f1af806ed70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1b08009bf0 tx=0x7f1b08009110 comp rx=0 tx=0).stop 2024-01-21T03:56:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 10 monclient: shutdown 2024-01-21T03:56:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 1 -- 172.21.15.7:0/4078058781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b0c142b90 msgr2=0x7f1b0c1da4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 1 --2- 172.21.15.7:0/4078058781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b0c142b90 0x7f1b0c1da4d0 secure :-1 s=READY pgs=5834 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400c690 tx=0x7f1b0400cb50 comp rx=0 tx=0).stop 2024-01-21T03:56:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 1 -- 172.21.15.7:0/4078058781 shutdown_connections 2024-01-21T03:56:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 1 --2- 172.21.15.7:0/4078058781 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7f1af806c8b0 0x7f1af806ed70 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 1 --2- 172.21.15.7:0/4078058781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b0c142b90 0x7f1b0c1da4d0 unknown :-1 s=CLOSED pgs=5834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 1 --2- 172.21.15.7:0/4078058781 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b0c0b12d0 0x7f1b0c13adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 1 --2- 172.21.15.7:0/4078058781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b0c06c980 0x7f1b0c13d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 1 -- 172.21.15.7:0/4078058781 >> 172.21.15.7:0/4078058781 conn(0x7f1b0c0b23c0 msgr2=0x7f1b0c071850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 1 -- 172.21.15.7:0/4078058781 shutdown_connections 2024-01-21T03:56:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 1 -- 172.21.15.7:0/4078058781 wait complete. 2024-01-21T03:56:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.048+0000 7f1b12d5e640 1 librados: shutdown 2024-01-21T03:56:57.059 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":2413,"stamp":"2024-01-21T03:56:56.238876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":600920,"kb_used_data":40600,"kb_used_omap":12,"kb_used_meta":560243,"kb_avail":754373800,"statfs":{"total":773094113280,"available":772478771200,"internally_reserved":0,"allocated":41574400,"data_stored":19456474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":573689393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002616"},"pg_stats":[{"pgid":"14.1e","version":"0'0","reported_seq":1264,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:38.296792+0000","last_change":"2024-01-21T03:56:38.298132+0000","last_active":"2024-01-21T03:56:38.296792+0000","last_peered":"2024-01-21T03:56:38.296792+0000","last_clean":"2024-01-21T03:56:38.296792+0000","last_became_active":"2024-01-21T03:21:15.289479+0000","last_became_peered":"2024-01-21T03:21:15.289479+0000","last_unstale":"2024-01-21T03:56:38.296792+0000","last_undegraded":"2024-01-21T03:56:38.296792+0000","last_fullsized":"2024-01-21T03:56:38.296792+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:38.294852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:38.294852+0000","last_clean_scrub_stamp":"2024-01-21T03:56:38.294852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:42:15.524369+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1223,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:37.649749+0000","last_change":"2024-01-21T03:56:37.650536+0000","last_active":"2024-01-21T03:56:37.649749+0000","last_peered":"2024-01-21T03:56:37.649749+0000","last_clean":"2024-01-21T03:56:37.649749+0000","last_became_active":"2024-01-21T03:21:15.864718+0000","last_became_peered":"2024-01-21T03:21:15.864718+0000","last_unstale":"2024-01-21T03:56:37.649749+0000","last_undegraded":"2024-01-21T03:56:37.649749+0000","last_fullsized":"2024-01-21T03:56:37.649749+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:37.648299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:37.648299+0000","last_clean_scrub_stamp":"2024-01-21T03:56:37.648299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:09:54.881027+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1244,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:35.250694+0000","last_change":"2024-01-21T03:56:35.251432+0000","last_active":"2024-01-21T03:56:35.250694+0000","last_peered":"2024-01-21T03:56:35.250694+0000","last_clean":"2024-01-21T03:56:35.250694+0000","last_became_active":"2024-01-21T03:21:09.333873+0000","last_became_peered":"2024-01-21T03:21:09.333873+0000","last_unstale":"2024-01-21T03:56:35.250694+0000","last_undegraded":"2024-01-21T03:56:35.250694+0000","last_fullsized":"2024-01-21T03:56:35.250694+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:35.249282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:35.249282+0000","last_clean_scrub_stamp":"2024-01-21T03:56:35.249282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:05:55.058446+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1241,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:35.367975+0000","last_change":"2024-01-21T03:56:35.368687+0000","last_active":"2024-01-21T03:56:35.367975+0000","last_peered":"2024-01-21T03:56:35.367975+0000","last_clean":"2024-01-21T03:56:35.367975+0000","last_became_active":"2024-01-21T03:21:13.539461+0000","last_became_peered":"2024-01-21T03:21:13.539461+0000","last_unstale":"2024-01-21T03:56:35.367975+0000","last_undegraded":"2024-01-21T03:56:35.367975+0000","last_fullsized":"2024-01-21T03:56:35.367975+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:35.366661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:35.366661+0000","last_clean_scrub_stamp":"2024-01-21T03:56:35.366661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:39:37.439688+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1221,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:33.133783+0000","last_change":"2024-01-21T03:56:33.133783+0000","last_active":"2024-01-21T03:56:33.133783+0000","last_peered":"2024-01-21T03:56:33.133783+0000","last_clean":"2024-01-21T03:56:33.133783+0000","last_became_active":"2024-01-21T03:21:30.818768+0000","last_became_peered":"2024-01-21T03:21:30.818768+0000","last_unstale":"2024-01-21T03:56:33.133783+0000","last_undegraded":"2024-01-21T03:56:33.133783+0000","last_fullsized":"2024-01-21T03:56:33.133783+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:33.131559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:33.131559+0000","last_clean_scrub_stamp":"2024-01-21T03:56:33.131559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:22:07.579587+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1244,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:36.430150+0000","last_change":"2024-01-21T03:56:36.430150+0000","last_active":"2024-01-21T03:56:36.430150+0000","last_peered":"2024-01-21T03:56:36.430150+0000","last_clean":"2024-01-21T03:56:36.430150+0000","last_became_active":"2024-01-21T03:21:13.842398+0000","last_became_peered":"2024-01-21T03:21:13.842398+0000","last_unstale":"2024-01-21T03:56:36.430150+0000","last_undegraded":"2024-01-21T03:56:36.430150+0000","last_fullsized":"2024-01-21T03:56:36.430150+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:36.428012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:36.428012+0000","last_clean_scrub_stamp":"2024-01-21T03:56:36.428012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:00:06.449073+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1244,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:38.377358+0000","last_change":"2024-01-21T03:56:38.378071+0000","last_active":"2024-01-21T03:56:38.377358+0000","last_peered":"2024-01-21T03:56:38.377358+0000","last_clean":"2024-01-21T03:56:38.377358+0000","last_became_active":"2024-01-21T03:21:34.290896+0000","last_became_peered":"2024-01-21T03:21:34.290896+0000","last_unstale":"2024-01-21T03:56:38.377358+0000","last_undegraded":"2024-01-21T03:56:38.377358+0000","last_fullsized":"2024-01-21T03:56:38.377358+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:38.376029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:38.376029+0000","last_clean_scrub_stamp":"2024-01-21T03:56:38.376029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:17:18.916647+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1","version":"1294'8","reported_seq":1238,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:38.776431+0000","last_change":"2024-01-21T03:56:38.777775+0000","last_active":"2024-01-21T03:56:38.776431+0000","last_peered":"2024-01-21T03:56:38.776431+0000","last_clean":"2024-01-21T03:56:38.776431+0000","last_became_active":"2024-01-21T03:21:35.763549+0000","last_became_peered":"2024-01-21T03:21:35.763549+0000","last_unstale":"2024-01-21T03:56:38.776431+0000","last_undegraded":"2024-01-21T03:56:38.776431+0000","last_fullsized":"2024-01-21T03:56:38.776431+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"1294'8","last_scrub_stamp":"2024-01-21T03:56:38.775082+0000","last_deep_scrub":"1294'8","last_deep_scrub_stamp":"2024-01-21T03:56:38.775082+0000","last_clean_scrub_stamp":"2024-01-21T03:56:38.775082+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:29:48.795599+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1033,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:31.457626+0000","last_change":"2024-01-21T03:56:31.458427+0000","last_active":"2024-01-21T03:56:31.457626+0000","last_peered":"2024-01-21T03:56:31.457626+0000","last_clean":"2024-01-21T03:56:31.457626+0000","last_became_active":"2024-01-21T03:26:06.758343+0000","last_became_peered":"2024-01-21T03:26:06.758343+0000","last_unstale":"2024-01-21T03:56:31.457626+0000","last_undegraded":"2024-01-21T03:56:31.457626+0000","last_fullsized":"2024-01-21T03:56:31.457626+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:31.456183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:31.456183+0000","last_clean_scrub_stamp":"2024-01-21T03:56:31.456183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:29:15.558115+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1239,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:33.005520+0000","last_change":"2024-01-21T03:56:33.006322+0000","last_active":"2024-01-21T03:56:33.005520+0000","last_peered":"2024-01-21T03:56:33.005520+0000","last_clean":"2024-01-21T03:56:33.005520+0000","last_became_active":"2024-01-21T03:21:15.779865+0000","last_became_peered":"2024-01-21T03:21:15.779865+0000","last_unstale":"2024-01-21T03:56:33.005520+0000","last_undegraded":"2024-01-21T03:56:33.005520+0000","last_fullsized":"2024-01-21T03:56:33.005520+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:33.004104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:33.004104+0000","last_clean_scrub_stamp":"2024-01-21T03:56:33.004104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:52:31.041691+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1221,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:32.167146+0000","last_change":"2024-01-21T03:56:32.167146+0000","last_active":"2024-01-21T03:56:32.167146+0000","last_peered":"2024-01-21T03:56:32.167146+0000","last_clean":"2024-01-21T03:56:32.167146+0000","last_became_active":"2024-01-21T03:21:22.647696+0000","last_became_peered":"2024-01-21T03:21:22.647696+0000","last_unstale":"2024-01-21T03:56:32.167146+0000","last_undegraded":"2024-01-21T03:56:32.167146+0000","last_fullsized":"2024-01-21T03:56:32.167146+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:32.164849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:32.164849+0000","last_clean_scrub_stamp":"2024-01-21T03:56:32.164849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:48:51.806532+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1232,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:38.323095+0000","last_change":"2024-01-21T03:56:38.323095+0000","last_active":"2024-01-21T03:56:38.323095+0000","last_peered":"2024-01-21T03:56:38.323095+0000","last_clean":"2024-01-21T03:56:38.323095+0000","last_became_active":"2024-01-21T03:21:26.223079+0000","last_became_peered":"2024-01-21T03:21:26.223079+0000","last_unstale":"2024-01-21T03:56:38.323095+0000","last_undegraded":"2024-01-21T03:56:38.323095+0000","last_fullsized":"2024-01-21T03:56:38.323095+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:38.320273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:38.320273+0000","last_clean_scrub_stamp":"2024-01-21T03:56:38.320273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:25:52.594394+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1245,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:36.115236+0000","last_change":"2024-01-21T03:56:36.116035+0000","last_active":"2024-01-21T03:56:36.115236+0000","last_peered":"2024-01-21T03:56:36.115236+0000","last_clean":"2024-01-21T03:56:36.115236+0000","last_became_active":"2024-01-21T03:26:06.693281+0000","last_became_peered":"2024-01-21T03:26:06.693281+0000","last_unstale":"2024-01-21T03:56:36.115236+0000","last_undegraded":"2024-01-21T03:56:36.115236+0000","last_fullsized":"2024-01-21T03:56:36.115236+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:36.113807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:36.113807+0000","last_clean_scrub_stamp":"2024-01-21T03:56:36.113807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:46:43.150533+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1294'78","reported_seq":1272,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:30.986195+0000","last_change":"2024-01-21T03:56:30.987756+0000","last_active":"2024-01-21T03:56:30.986195+0000","last_peered":"2024-01-21T03:56:30.986195+0000","last_clean":"2024-01-21T03:56:30.986195+0000","last_became_active":"2024-01-21T03:19:58.060095+0000","last_became_peered":"2024-01-21T03:19:58.060095+0000","last_unstale":"2024-01-21T03:56:30.986195+0000","last_undegraded":"2024-01-21T03:56:30.986195+0000","last_fullsized":"2024-01-21T03:56:30.986195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"1294'78","last_scrub_stamp":"2024-01-21T03:56:30.981858+0000","last_deep_scrub":"1294'78","last_deep_scrub_stamp":"2024-01-21T03:56:30.981858+0000","last_clean_scrub_stamp":"2024-01-21T03:56:30.981858+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:38:53.709518+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1270,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:37.229119+0000","last_change":"2024-01-21T03:56:37.230381+0000","last_active":"2024-01-21T03:56:37.229119+0000","last_peered":"2024-01-21T03:56:37.229119+0000","last_clean":"2024-01-21T03:56:37.229119+0000","last_became_active":"2024-01-21T03:20:02.289915+0000","last_became_peered":"2024-01-21T03:20:02.289915+0000","last_unstale":"2024-01-21T03:56:37.229119+0000","last_undegraded":"2024-01-21T03:56:37.229119+0000","last_fullsized":"2024-01-21T03:56:37.229119+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:37.227534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:37.227534+0000","last_clean_scrub_stamp":"2024-01-21T03:56:37.227534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:14:55.668005+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1231,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:37.270789+0000","last_change":"2024-01-21T03:56:37.271564+0000","last_active":"2024-01-21T03:56:37.270789+0000","last_peered":"2024-01-21T03:56:37.270789+0000","last_clean":"2024-01-21T03:56:37.270789+0000","last_became_active":"2024-01-21T03:21:14.829810+0000","last_became_peered":"2024-01-21T03:21:14.829810+0000","last_unstale":"2024-01-21T03:56:37.270789+0000","last_undegraded":"2024-01-21T03:56:37.270789+0000","last_fullsized":"2024-01-21T03:56:37.270789+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:37.269379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:37.269379+0000","last_clean_scrub_stamp":"2024-01-21T03:56:37.269379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:58:01.845590+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1271,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:33.190367+0000","last_change":"2024-01-21T03:56:33.191175+0000","last_active":"2024-01-21T03:56:33.190367+0000","last_peered":"2024-01-21T03:56:33.190367+0000","last_clean":"2024-01-21T03:56:33.190367+0000","last_became_active":"2024-01-21T03:20:01.784281+0000","last_became_peered":"2024-01-21T03:20:01.784281+0000","last_unstale":"2024-01-21T03:56:33.190367+0000","last_undegraded":"2024-01-21T03:56:33.190367+0000","last_fullsized":"2024-01-21T03:56:33.190367+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:33.188920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:33.188920+0000","last_clean_scrub_stamp":"2024-01-21T03:56:33.188920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:14:38.519289+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1264,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:39.310907+0000","last_change":"2024-01-21T03:56:39.311699+0000","last_active":"2024-01-21T03:56:39.310907+0000","last_peered":"2024-01-21T03:56:39.310907+0000","last_clean":"2024-01-21T03:56:39.310907+0000","last_became_active":"2024-01-21T03:21:26.057996+0000","last_became_peered":"2024-01-21T03:21:26.057996+0000","last_unstale":"2024-01-21T03:56:39.310907+0000","last_undegraded":"2024-01-21T03:56:39.310907+0000","last_fullsized":"2024-01-21T03:56:39.310907+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:39.309484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:39.309484+0000","last_clean_scrub_stamp":"2024-01-21T03:56:39.309484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:28:21.474362+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1228,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:36.771063+0000","last_change":"2024-01-21T03:56:36.775623+0000","last_active":"2024-01-21T03:56:36.771063+0000","last_peered":"2024-01-21T03:56:36.771063+0000","last_clean":"2024-01-21T03:56:36.771063+0000","last_became_active":"2024-01-21T03:20:01.818592+0000","last_became_peered":"2024-01-21T03:20:01.818592+0000","last_unstale":"2024-01-21T03:56:36.771063+0000","last_undegraded":"2024-01-21T03:56:36.771063+0000","last_fullsized":"2024-01-21T03:56:36.771063+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:36.766968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:36.766968+0000","last_clean_scrub_stamp":"2024-01-21T03:56:36.766968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:46:25.340129+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1239,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:33.981328+0000","last_change":"2024-01-21T03:56:33.982126+0000","last_active":"2024-01-21T03:56:33.981328+0000","last_peered":"2024-01-21T03:56:33.981328+0000","last_clean":"2024-01-21T03:56:33.981328+0000","last_became_active":"2024-01-21T03:22:06.040269+0000","last_became_peered":"2024-01-21T03:22:06.040269+0000","last_unstale":"2024-01-21T03:56:33.981328+0000","last_undegraded":"2024-01-21T03:56:33.981328+0000","last_fullsized":"2024-01-21T03:56:33.981328+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:33.979724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:33.979724+0000","last_clean_scrub_stamp":"2024-01-21T03:56:33.979724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:01:51.141758+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1270,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:35.250508+0000","last_change":"2024-01-21T03:56:35.251321+0000","last_active":"2024-01-21T03:56:35.250508+0000","last_peered":"2024-01-21T03:56:35.250508+0000","last_clean":"2024-01-21T03:56:35.250508+0000","last_became_active":"2024-01-21T03:20:02.201314+0000","last_became_peered":"2024-01-21T03:20:02.201314+0000","last_unstale":"2024-01-21T03:56:35.250508+0000","last_undegraded":"2024-01-21T03:56:35.250508+0000","last_fullsized":"2024-01-21T03:56:35.250508+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:35.248863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:35.248863+0000","last_clean_scrub_stamp":"2024-01-21T03:56:35.248863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:46:21.364120+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1243,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:39.396276+0000","last_change":"2024-01-21T03:56:39.397058+0000","last_active":"2024-01-21T03:56:39.396276+0000","last_peered":"2024-01-21T03:56:39.396276+0000","last_clean":"2024-01-21T03:56:39.396276+0000","last_became_active":"2024-01-21T03:21:34.307508+0000","last_became_peered":"2024-01-21T03:21:34.307508+0000","last_unstale":"2024-01-21T03:56:39.396276+0000","last_undegraded":"2024-01-21T03:56:39.396276+0000","last_fullsized":"2024-01-21T03:56:39.396276+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:39.392189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:39.392189+0000","last_clean_scrub_stamp":"2024-01-21T03:56:39.392189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:32:29.782263+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1237,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:34.337634+0000","last_change":"2024-01-21T03:56:34.338430+0000","last_active":"2024-01-21T03:56:34.337634+0000","last_peered":"2024-01-21T03:56:34.337634+0000","last_clean":"2024-01-21T03:56:34.337634+0000","last_became_active":"2024-01-21T03:20:02.207814+0000","last_became_peered":"2024-01-21T03:20:02.207814+0000","last_unstale":"2024-01-21T03:56:34.337634+0000","last_undegraded":"2024-01-21T03:56:34.337634+0000","last_fullsized":"2024-01-21T03:56:34.337634+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:34.334201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:34.334201+0000","last_clean_scrub_stamp":"2024-01-21T03:56:34.334201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:34:11.411502+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1224,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:30.648133+0000","last_change":"2024-01-21T03:56:30.652891+0000","last_active":"2024-01-21T03:56:30.648133+0000","last_peered":"2024-01-21T03:56:30.648133+0000","last_clean":"2024-01-21T03:56:30.648133+0000","last_became_active":"2024-01-21T03:21:23.767027+0000","last_became_peered":"2024-01-21T03:21:23.767027+0000","last_unstale":"2024-01-21T03:56:30.648133+0000","last_undegraded":"2024-01-21T03:56:30.648133+0000","last_fullsized":"2024-01-21T03:56:30.648133+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:30.644030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:30.644030+0000","last_clean_scrub_stamp":"2024-01-21T03:56:30.644030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:43:36.949817+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1270,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:34.201029+0000","last_change":"2024-01-21T03:56:34.201835+0000","last_active":"2024-01-21T03:56:34.201029+0000","last_peered":"2024-01-21T03:56:34.201029+0000","last_clean":"2024-01-21T03:56:34.201029+0000","last_became_active":"2024-01-21T03:20:01.872724+0000","last_became_peered":"2024-01-21T03:20:01.872724+0000","last_unstale":"2024-01-21T03:56:34.201029+0000","last_undegraded":"2024-01-21T03:56:34.201029+0000","last_fullsized":"2024-01-21T03:56:34.201029+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:34.199531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:34.199531+0000","last_clean_scrub_stamp":"2024-01-21T03:56:34.199531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:24:00.875702+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1220,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:34.133980+0000","last_change":"2024-01-21T03:56:34.134770+0000","last_active":"2024-01-21T03:56:34.133980+0000","last_peered":"2024-01-21T03:56:34.133980+0000","last_clean":"2024-01-21T03:56:34.133980+0000","last_became_active":"2024-01-21T03:21:55.460120+0000","last_became_peered":"2024-01-21T03:21:55.460120+0000","last_unstale":"2024-01-21T03:56:34.133980+0000","last_undegraded":"2024-01-21T03:56:34.133980+0000","last_fullsized":"2024-01-21T03:56:34.133980+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:34.132582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:34.132582+0000","last_clean_scrub_stamp":"2024-01-21T03:56:34.132582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:59:06.058239+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1248,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:33.437223+0000","last_change":"2024-01-21T03:56:33.437223+0000","last_active":"2024-01-21T03:56:33.437223+0000","last_peered":"2024-01-21T03:56:33.437223+0000","last_clean":"2024-01-21T03:56:33.437223+0000","last_became_active":"2024-01-21T03:20:02.234092+0000","last_became_peered":"2024-01-21T03:20:02.234092+0000","last_unstale":"2024-01-21T03:56:33.437223+0000","last_undegraded":"2024-01-21T03:56:33.437223+0000","last_fullsized":"2024-01-21T03:56:33.437223+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:33.435060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:33.435060+0000","last_clean_scrub_stamp":"2024-01-21T03:56:33.435060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:27:57.141698+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1245,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:36.350838+0000","last_change":"2024-01-21T03:56:36.351598+0000","last_active":"2024-01-21T03:56:36.350838+0000","last_peered":"2024-01-21T03:56:36.350838+0000","last_clean":"2024-01-21T03:56:36.350838+0000","last_became_active":"2024-01-21T03:21:09.531293+0000","last_became_peered":"2024-01-21T03:21:09.531293+0000","last_unstale":"2024-01-21T03:56:36.350838+0000","last_undegraded":"2024-01-21T03:56:36.350838+0000","last_fullsized":"2024-01-21T03:56:36.350838+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:36.349433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:36.349433+0000","last_clean_scrub_stamp":"2024-01-21T03:56:36.349433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:06:58.828201+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1266,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:36.248761+0000","last_change":"2024-01-21T03:56:36.249564+0000","last_active":"2024-01-21T03:56:36.248761+0000","last_peered":"2024-01-21T03:56:36.248761+0000","last_clean":"2024-01-21T03:56:36.248761+0000","last_became_active":"2024-01-21T03:20:02.226975+0000","last_became_peered":"2024-01-21T03:20:02.226975+0000","last_unstale":"2024-01-21T03:56:36.248761+0000","last_undegraded":"2024-01-21T03:56:36.248761+0000","last_fullsized":"2024-01-21T03:56:36.248761+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:36.247256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:36.247256+0000","last_clean_scrub_stamp":"2024-01-21T03:56:36.247256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:36:21.969900+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1263,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:40.267697+0000","last_change":"2024-01-21T03:56:40.268497+0000","last_active":"2024-01-21T03:56:40.267697+0000","last_peered":"2024-01-21T03:56:40.267697+0000","last_clean":"2024-01-21T03:56:40.267697+0000","last_became_active":"2024-01-21T03:21:36.929227+0000","last_became_peered":"2024-01-21T03:21:36.929227+0000","last_unstale":"2024-01-21T03:56:40.267697+0000","last_undegraded":"2024-01-21T03:56:40.267697+0000","last_fullsized":"2024-01-21T03:56:40.267697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:40.266293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:40.266293+0000","last_clean_scrub_stamp":"2024-01-21T03:56:40.266293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:44:53.215521+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1231,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:35.344347+0000","last_change":"2024-01-21T03:56:35.345078+0000","last_active":"2024-01-21T03:56:35.344347+0000","last_peered":"2024-01-21T03:56:35.344347+0000","last_clean":"2024-01-21T03:56:35.344347+0000","last_became_active":"2024-01-21T03:21:13.969066+0000","last_became_peered":"2024-01-21T03:21:13.969066+0000","last_unstale":"2024-01-21T03:56:35.344347+0000","last_undegraded":"2024-01-21T03:56:35.344347+0000","last_fullsized":"2024-01-21T03:56:35.344347+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:35.340749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:35.340749+0000","last_clean_scrub_stamp":"2024-01-21T03:56:35.340749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:08:16.207579+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1244,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:38.346134+0000","last_change":"2024-01-21T03:56:38.346896+0000","last_active":"2024-01-21T03:56:38.346134+0000","last_peered":"2024-01-21T03:56:38.346134+0000","last_clean":"2024-01-21T03:56:38.346134+0000","last_became_active":"2024-01-21T03:21:22.319120+0000","last_became_peered":"2024-01-21T03:21:22.319120+0000","last_unstale":"2024-01-21T03:56:38.346134+0000","last_undegraded":"2024-01-21T03:56:38.346134+0000","last_fullsized":"2024-01-21T03:56:38.346134+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:38.344772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:38.344772+0000","last_clean_scrub_stamp":"2024-01-21T03:56:38.344772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:02:53.485024+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1241,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:37.357786+0000","last_change":"2024-01-21T03:56:37.358502+0000","last_active":"2024-01-21T03:56:37.357786+0000","last_peered":"2024-01-21T03:56:37.357786+0000","last_clean":"2024-01-21T03:56:37.357786+0000","last_became_active":"2024-01-21T03:21:17.582134+0000","last_became_peered":"2024-01-21T03:21:17.582134+0000","last_unstale":"2024-01-21T03:56:37.357786+0000","last_undegraded":"2024-01-21T03:56:37.357786+0000","last_fullsized":"2024-01-21T03:56:37.357786+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:37.356466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:37.356466+0000","last_clean_scrub_stamp":"2024-01-21T03:56:37.356466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:09:55.511014+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1263,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:41.242533+0000","last_change":"2024-01-21T03:56:41.243341+0000","last_active":"2024-01-21T03:56:41.242533+0000","last_peered":"2024-01-21T03:56:41.242533+0000","last_clean":"2024-01-21T03:56:41.242533+0000","last_became_active":"2024-01-21T03:21:40.568805+0000","last_became_peered":"2024-01-21T03:21:40.568805+0000","last_unstale":"2024-01-21T03:56:41.242533+0000","last_undegraded":"2024-01-21T03:56:41.242533+0000","last_fullsized":"2024-01-21T03:56:41.242533+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:41.241063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:41.241063+0000","last_clean_scrub_stamp":"2024-01-21T03:56:41.241063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:57:44.295461+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1239,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:35.006906+0000","last_change":"2024-01-21T03:56:35.008290+0000","last_active":"2024-01-21T03:56:35.006906+0000","last_peered":"2024-01-21T03:56:35.006906+0000","last_clean":"2024-01-21T03:56:35.006906+0000","last_became_active":"2024-01-21T03:22:06.149457+0000","last_became_peered":"2024-01-21T03:22:06.149457+0000","last_unstale":"2024-01-21T03:56:35.006906+0000","last_undegraded":"2024-01-21T03:56:35.006906+0000","last_fullsized":"2024-01-21T03:56:35.006906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:35.002390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:35.002390+0000","last_clean_scrub_stamp":"2024-01-21T03:56:35.002390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:32:10.186897+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1241,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:34.439810+0000","last_change":"2024-01-21T03:56:34.440574+0000","last_active":"2024-01-21T03:56:34.439810+0000","last_peered":"2024-01-21T03:56:34.439810+0000","last_clean":"2024-01-21T03:56:34.439810+0000","last_became_active":"2024-01-21T03:21:13.373828+0000","last_became_peered":"2024-01-21T03:21:13.373828+0000","last_unstale":"2024-01-21T03:56:34.439810+0000","last_undegraded":"2024-01-21T03:56:34.439810+0000","last_fullsized":"2024-01-21T03:56:34.439810+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:34.438416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:34.438416+0000","last_clean_scrub_stamp":"2024-01-21T03:56:34.438416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:26:45.049747+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1242,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:37.269601+0000","last_change":"2024-01-21T03:56:37.270381+0000","last_active":"2024-01-21T03:56:37.269601+0000","last_peered":"2024-01-21T03:56:37.269601+0000","last_clean":"2024-01-21T03:56:37.269601+0000","last_became_active":"2024-01-21T03:21:10.999264+0000","last_became_peered":"2024-01-21T03:21:10.999264+0000","last_unstale":"2024-01-21T03:56:37.269601+0000","last_undegraded":"2024-01-21T03:56:37.269601+0000","last_fullsized":"2024-01-21T03:56:37.269601+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:37.268181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:37.268181+0000","last_clean_scrub_stamp":"2024-01-21T03:56:37.268181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:53:40.711997+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1239,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:31.922554+0000","last_change":"2024-01-21T03:56:31.923364+0000","last_active":"2024-01-21T03:56:31.922554+0000","last_peered":"2024-01-21T03:56:31.922554+0000","last_clean":"2024-01-21T03:56:31.922554+0000","last_became_active":"2024-01-21T03:21:15.068960+0000","last_became_peered":"2024-01-21T03:21:15.068960+0000","last_unstale":"2024-01-21T03:56:31.922554+0000","last_undegraded":"2024-01-21T03:56:31.922554+0000","last_fullsized":"2024-01-21T03:56:31.922554+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:31.921105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:31.921105+0000","last_clean_scrub_stamp":"2024-01-21T03:56:31.921105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:56:24.462551+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1245,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:39.307424+0000","last_change":"2024-01-21T03:56:39.308143+0000","last_active":"2024-01-21T03:56:39.307424+0000","last_peered":"2024-01-21T03:56:39.307424+0000","last_clean":"2024-01-21T03:56:39.307424+0000","last_became_active":"2024-01-21T03:21:32.997061+0000","last_became_peered":"2024-01-21T03:21:32.997061+0000","last_unstale":"2024-01-21T03:56:39.307424+0000","last_undegraded":"2024-01-21T03:56:39.307424+0000","last_fullsized":"2024-01-21T03:56:39.307424+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:39.306134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:39.306134+0000","last_clean_scrub_stamp":"2024-01-21T03:56:39.306134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:47:27.145822+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1231,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:36.366764+0000","last_change":"2024-01-21T03:56:36.367544+0000","last_active":"2024-01-21T03:56:36.366764+0000","last_peered":"2024-01-21T03:56:36.366764+0000","last_clean":"2024-01-21T03:56:36.366764+0000","last_became_active":"2024-01-21T03:21:14.761665+0000","last_became_peered":"2024-01-21T03:21:14.761665+0000","last_unstale":"2024-01-21T03:56:36.366764+0000","last_undegraded":"2024-01-21T03:56:36.366764+0000","last_fullsized":"2024-01-21T03:56:36.366764+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:36.365328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:36.365328+0000","last_clean_scrub_stamp":"2024-01-21T03:56:36.365328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:47:52.035050+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1245,"reported_epoch":1294,"state":"active+clean","last_fresh":"2024-01-21T03:56:40.318330+0000","last_change":"2024-01-21T03:56:40.319292+0000","last_active":"2024-01-21T03:56:40.318330+0000","last_peered":"2024-01-21T03:56:40.318330+0000","last_clean":"2024-01-21T03:56:40.318330+0000","last_became_active":"2024-01-21T03:21:34.168272+0000","last_became_peered":"2024-01-21T03:21:34.168272+0000","last_unstale":"2024-01-21T03:56:40.318330+0000","last_undegraded":"2024-01-21T03:56:40.318330+0000","last_fullsized":"2024-01-21T03:56:40.318330+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T03:56:40.316865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T03:56:40.316865+0000","last_clean_scrub_stamp":"2024-01-21T03:56:40.316865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T03:59:02.922911+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280954,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76408,"kb_used_data":4976,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295432,"statfs":{"total":96636764160,"available":96558522368,"internally_reserved":0,"allocated":5095424,"data_stored":2332598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":9.3070000000000004,"5min":22.673999999999999,"15min":23.361000000000001},"min":{"1min":5.9930000000000003,"5min":4.7569999999999997,"15min":4.2290000000000001},"max":{"1min":15.645,"5min":221.726,"15min":236.40199999999999},"last":9.5600000000000005},{"interface":"front","average":{"1min":12.923,"5min":28.297999999999998,"15min":22.507000000000001},"min":{"1min":7.3280000000000003,"5min":6.5869999999999997,"15min":4.423},"max":{"1min":21.279,"5min":315.84899999999999,"15min":317.16500000000002},"last":18.016999999999999}]},{"osd":1,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":12.68,"5min":24.010999999999999,"15min":26.009},"min":{"1min":7.8280000000000003,"5min":7.8280000000000003,"15min":6.7649999999999997},"max":{"1min":19.681999999999999,"5min":210.08799999999999,"15min":688.34500000000003},"last":8.8870000000000005},{"interface":"front","average":{"1min":15.066000000000001,"5min":30.968,"15min":31.401},"min":{"1min":10.843999999999999,"5min":10.49,"15min":9.6310000000000002},"max":{"1min":23.169,"5min":288.46600000000001,"15min":686.94299999999998},"last":11.875999999999999}]},{"osd":2,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":9.9570000000000007,"5min":59.222999999999999,"15min":34.966999999999999},"min":{"1min":6.0730000000000004,"5min":6.0730000000000004,"15min":5.75},"max":{"1min":31.401,"5min":3500.2379999999998,"15min":3500.2379999999998},"last":14.007},{"interface":"front","average":{"1min":14.031000000000001,"5min":37.201000000000001,"15min":29.867000000000001},"min":{"1min":7.9669999999999996,"5min":7.9669999999999996,"15min":7.4710000000000001},"max":{"1min":25.942,"5min":919.58199999999999,"15min":919.58199999999999},"last":14.648999999999999}]},{"osd":4,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":14.010999999999999,"5min":26.649000000000001,"15min":26.817},"min":{"1min":10.026,"5min":9.1489999999999991,"15min":8.9909999999999997},"max":{"1min":18.762,"5min":241.95599999999999,"15min":297.46600000000001},"last":7.0110000000000001},{"interface":"front","average":{"1min":18.001999999999999,"5min":31.356999999999999,"15min":31.646999999999998},"min":{"1min":12.919,"5min":12.346,"15min":9.673},"max":{"1min":23.687000000000001,"5min":292.25400000000002,"15min":340.73500000000001},"last":12.547000000000001}]},{"osd":5,"last update":"Sun Jan 21 03:56:18 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":24.042999999999999,"15min":24.727},"min":{"1min":7.3680000000000003,"5min":7.3680000000000003,"15min":6.1989999999999998},"max":{"1min":22.300000000000001,"5min":214.58500000000001,"15min":362.22899999999998},"last":15.304},{"interface":"front","average":{"1min":13.831,"5min":26.408000000000001,"15min":27.535},"min":{"1min":8.9079999999999995,"5min":6.5439999999999996,"15min":6.5439999999999996},"max":{"1min":21.576000000000001,"5min":339.19499999999999,"15min":339.19499999999999},"last":10.302}]},{"osd":6,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":18.814,"5min":31.257000000000001,"15min":30.303999999999998},"min":{"1min":11.645,"5min":11.206,"15min":7.1719999999999997},"max":{"1min":42.293999999999997,"5min":211.822,"15min":572.20899999999995},"last":8.1519999999999992},{"interface":"front","average":{"1min":21.713000000000001,"5min":36.789999999999999,"15min":34.954999999999998},"min":{"1min":13.535,"5min":13.535,"15min":9.7240000000000002},"max":{"1min":45.457999999999998,"5min":215.315,"15min":376.66399999999999},"last":13.223000000000001}]},{"osd":7,"last update":"Sun Jan 21 03:56:18 2024","interfaces":[{"interface":"back","average":{"1min":15.071999999999999,"5min":23.681999999999999,"15min":23.119},"min":{"1min":9.3960000000000008,"5min":8.1329999999999991,"15min":5.9169999999999998},"max":{"1min":30.167999999999999,"5min":213.73500000000001,"15min":274.91000000000003},"last":18.68},{"interface":"front","average":{"1min":21.024000000000001,"5min":32.463999999999999,"15min":32.999000000000002},"min":{"1min":14.153,"5min":11.337,"15min":11.337},"max":{"1min":31.466000000000001,"5min":234.46299999999999,"15min":339.613},"last":11.225}]}]},{"osd":2,"up_from":21,"seq":90194313662,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68088,"kb_used_data":4976,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303752,"statfs":{"total":96636764160,"available":96567042048,"internally_reserved":0,"allocated":5095424,"data_stored":2332598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:55:51 2024","interfaces":[{"interface":"back","average":{"1min":11.087,"5min":20.286999999999999,"15min":17.802},"min":{"1min":4.9820000000000002,"5min":4.9820000000000002,"15min":4.5460000000000003},"max":{"1min":23.617000000000001,"5min":335.38200000000001,"15min":335.38200000000001},"last":9.8510000000000009},{"interface":"front","average":{"1min":9.5939999999999994,"5min":20.925999999999998,"15min":17.704000000000001},"min":{"1min":5.484,"5min":5.484,"15min":4.4580000000000002},"max":{"1min":22.068999999999999,"5min":343.05599999999998,"15min":343.05599999999998},"last":11.492000000000001}]},{"osd":1,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":16.661999999999999,"5min":32.128,"15min":25.600999999999999},"min":{"1min":7.641,"5min":5.5960000000000001,"15min":5.2190000000000003},"max":{"1min":50.104999999999997,"5min":341.34399999999999,"15min":504.58100000000002},"last":12.345000000000001},{"interface":"front","average":{"1min":14.928000000000001,"5min":35.445,"15min":27.994},"min":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":5.7350000000000003},"max":{"1min":32.649000000000001,"5min":674.13400000000001,"15min":674.13400000000001},"last":15.127000000000001}]},{"osd":3,"last update":"Sun Jan 21 03:56:39 2024","interfaces":[{"interface":"back","average":{"1min":19.856999999999999,"5min":21.544,"15min":19.306999999999999},"min":{"1min":7.6799999999999997,"5min":6.9950000000000001,"15min":6.4930000000000003},"max":{"1min":48.454000000000001,"5min":154.56299999999999,"15min":263.31400000000002},"last":8.7129999999999992},{"interface":"front","average":{"1min":20.652000000000001,"5min":27.312999999999999,"15min":21.491},"min":{"1min":12.577,"5min":8.9060000000000006,"15min":8.1180000000000003},"max":{"1min":33.831000000000003,"5min":683.75800000000004,"15min":683.75800000000004},"last":16.949999999999999}]},{"osd":4,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":22.283000000000001,"5min":29.207999999999998,"15min":23.219999999999999},"min":{"1min":10.741,"5min":8.827,"15min":8.0609999999999999},"max":{"1min":68.728999999999999,"5min":687.15800000000002,"15min":687.15800000000002},"last":10.613},{"interface":"front","average":{"1min":26.204000000000001,"5min":26.994,"15min":21.989999999999998},"min":{"1min":11.539999999999999,"5min":8.8379999999999992,"15min":8.1620000000000008},"max":{"1min":108.23,"5min":318.77199999999999,"15min":318.77199999999999},"last":19.081}]},{"osd":5,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":19.896000000000001,"5min":30.462,"15min":24.710999999999999},"min":{"1min":7.9740000000000002,"5min":7.6660000000000004,"15min":6.0650000000000004},"max":{"1min":50.735999999999997,"5min":348.47000000000003,"15min":348.47000000000003},"last":16.120999999999999},{"interface":"front","average":{"1min":21.834,"5min":30.434999999999999,"15min":25.710000000000001},"min":{"1min":10.997999999999999,"5min":10.967000000000001,"15min":8.5210000000000008},"max":{"1min":52.564999999999998,"5min":264.41199999999998,"15min":264.41199999999998},"last":13.661}]},{"osd":6,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":21.768999999999998,"5min":32.344000000000001,"15min":25.484000000000002},"min":{"1min":13.74,"5min":8.5960000000000001,"15min":7.2800000000000002},"max":{"1min":34.429000000000002,"5min":679.51900000000001,"15min":679.51900000000001},"last":18.076000000000001},{"interface":"front","average":{"1min":24.733000000000001,"5min":35.838000000000001,"15min":26.829999999999998},"min":{"1min":15.788,"5min":11.081,"15min":10.121},"max":{"1min":37.802,"5min":676.11800000000005,"15min":676.11800000000005},"last":19.991}]},{"osd":7,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":24.338999999999999,"5min":29.204999999999998,"15min":25.914000000000001},"min":{"1min":12.472,"5min":10.278,"15min":10.278},"max":{"1min":53.186999999999998,"5min":261.48899999999998,"15min":261.48899999999998},"last":13.012},{"interface":"front","average":{"1min":24.440000000000001,"5min":38.350999999999999,"15min":27.709},"min":{"1min":15.018000000000001,"5min":12.605,"15min":8.4239999999999995},"max":{"1min":42.475000000000001,"5min":678.09500000000003,"15min":678.09500000000003},"last":20.911999999999999}]}]},{"osd":1,"up_from":21,"seq":90194313661,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94784,"kb_used_data":5368,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94277056,"statfs":{"total":96636764160,"available":96539705344,"internally_reserved":0,"allocated":5496832,"data_stored":2730438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":11.568,"5min":16.558,"15min":21.972999999999999},"min":{"1min":4.5970000000000004,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":26.126000000000001,"5min":154.52799999999999,"15min":528.255},"last":15.247},{"interface":"front","average":{"1min":12.198,"5min":15.057,"15min":19.050999999999998},"min":{"1min":4.9619999999999997,"5min":4.5199999999999996,"15min":4.3079999999999998},"max":{"1min":24.762,"5min":127.73999999999999,"15min":447.83199999999999},"last":7.1859999999999999}]},{"osd":2,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":13.651,"5min":17.736999999999998,"15min":23.350999999999999},"min":{"1min":8.5549999999999997,"5min":7.8719999999999999,"15min":6.194},"max":{"1min":21.834,"5min":116.48999999999999,"15min":420.87700000000001},"last":10.864000000000001},{"interface":"front","average":{"1min":15.859999999999999,"5min":26.047999999999998,"15min":27.696999999999999},"min":{"1min":8.5239999999999991,"5min":7.9889999999999999,"15min":7.3209999999999997},"max":{"1min":30.548999999999999,"5min":265.24200000000002,"15min":451.637},"last":10.199}]},{"osd":3,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":20.800000000000001,"5min":22.111000000000001,"15min":23.620999999999999},"min":{"1min":9.2140000000000004,"5min":8.0370000000000008,"15min":8.0370000000000008},"max":{"1min":62.801000000000002,"5min":141.52099999999999,"15min":419.892},"last":12.164},{"interface":"front","average":{"1min":28.942,"5min":25.436,"15min":27.707000000000001},"min":{"1min":8.9109999999999996,"5min":8.6479999999999997,"15min":8.3059999999999992},"max":{"1min":123.798,"5min":143.58699999999999,"15min":425.76100000000002},"last":14.615}]},{"osd":4,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":26.920999999999999,"5min":26.507999999999999,"15min":25.122},"min":{"1min":11.009,"5min":7.4210000000000003,"15min":7.4210000000000003},"max":{"1min":106.116,"5min":229.63200000000001,"15min":344.48200000000003},"last":11.542999999999999},{"interface":"front","average":{"1min":25.748999999999999,"5min":29.904,"15min":30.959},"min":{"1min":12.282,"5min":8.3059999999999992,"15min":8.1259999999999994},"max":{"1min":65.070999999999998,"5min":236.571,"15min":385.43000000000001},"last":17.385000000000002}]},{"osd":5,"last update":"Sun Jan 21 03:56:19 2024","interfaces":[{"interface":"back","average":{"1min":22.766999999999999,"5min":28.076000000000001,"15min":30.946000000000002},"min":{"1min":13.015000000000001,"5min":9.5839999999999996,"15min":9.5839999999999996},"max":{"1min":38.545999999999999,"5min":138.84899999999999,"15min":338.88200000000001},"last":19.193999999999999},{"interface":"front","average":{"1min":18.858000000000001,"5min":26.266999999999999,"15min":29.963999999999999},"min":{"1min":7.577,"5min":7.577,"15min":7.577},"max":{"1min":34.542999999999999,"5min":187.03,"15min":356.423},"last":13.859999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":21.745000000000001,"5min":31.524000000000001,"15min":31.725000000000001},"min":{"1min":10.369999999999999,"5min":10.369999999999999,"15min":9.7289999999999992},"max":{"1min":53.404000000000003,"5min":280.85000000000002,"15min":409.08100000000002},"last":15.92},{"interface":"front","average":{"1min":20.478000000000002,"5min":25.875,"15min":23.920000000000002},"min":{"1min":6.6100000000000003,"5min":6.6100000000000003,"15min":6.6100000000000003},"max":{"1min":59.128999999999998,"5min":243.083,"15min":261.75099999999998},"last":8.6869999999999994}]},{"osd":7,"last update":"Sun Jan 21 03:56:12 2024","interfaces":[{"interface":"back","average":{"1min":21.91,"5min":28.564,"15min":25.913},"min":{"1min":12.441000000000001,"5min":11.923,"15min":7.21},"max":{"1min":43.122999999999998,"5min":169.571,"15min":357.57299999999998},"last":16.597999999999999},{"interface":"front","average":{"1min":23.428000000000001,"5min":23.902999999999999,"15min":25.715},"min":{"1min":8.6929999999999996,"5min":8.1959999999999997,"15min":7.9729999999999999},"max":{"1min":63.999000000000002,"5min":176.809,"15min":418.70600000000002},"last":9.2959999999999994}]}]},{"osd":0,"up_from":21,"seq":90194313665,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72248,"kb_used_data":4976,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299592,"statfs":{"total":96636764160,"available":96562782208,"internally_reserved":0,"allocated":5095424,"data_stored":2332598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":12.715,"5min":25.683,"15min":21.652000000000001},"min":{"1min":6.1859999999999999,"5min":6.0220000000000002,"15min":5.7869999999999999},"max":{"1min":33.234999999999999,"5min":246.35400000000001,"15min":301.80599999999998},"last":5.8209999999999997},{"interface":"front","average":{"1min":17.094000000000001,"5min":28.422000000000001,"15min":24.061},"min":{"1min":8.625,"5min":6.0270000000000001,"15min":6.0270000000000001},"max":{"1min":33.862000000000002,"5min":232.851,"15min":242.80000000000001},"last":6.4640000000000004}]},{"osd":2,"last update":"Sun Jan 21 03:56:45 2024","interfaces":[{"interface":"back","average":{"1min":18.256,"5min":28.800999999999998,"15min":26.004000000000001},"min":{"1min":8.8879999999999999,"5min":7.2160000000000002,"15min":6.6289999999999996},"max":{"1min":32.616,"5min":279.71300000000002,"15min":312.15800000000002},"last":12.425000000000001},{"interface":"front","average":{"1min":12.025,"5min":28.818999999999999,"15min":25.460000000000001},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":5.0910000000000002},"max":{"1min":28.356000000000002,"5min":332.57499999999999,"15min":357.38900000000001},"last":10.523999999999999}]},{"osd":3,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":43.631,"15min":30.786999999999999},"min":{"1min":11.736000000000001,"5min":7.5049999999999999,"15min":6.9829999999999997},"max":{"1min":56.691000000000003,"5min":547.15800000000002,"15min":547.15800000000002},"last":13.058},{"interface":"front","average":{"1min":22.646000000000001,"5min":42.786000000000001,"15min":31.776},"min":{"1min":7.4199999999999999,"5min":7.4199999999999999,"15min":7.4199999999999999},"max":{"1min":98.448999999999998,"5min":361.11500000000001,"15min":361.11500000000001},"last":11.797000000000001}]},{"osd":4,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":18.094000000000001,"5min":35.396999999999998,"15min":25.135999999999999},"min":{"1min":10.619999999999999,"5min":7.4630000000000001,"15min":7.3019999999999996},"max":{"1min":30.510000000000002,"5min":333.23899999999998,"15min":333.23899999999998},"last":8.7040000000000006},{"interface":"front","average":{"1min":27.469000000000001,"5min":42.499000000000002,"15min":29.593},"min":{"1min":7.0999999999999996,"5min":7.0999999999999996,"15min":7.0999999999999996},"max":{"1min":117.73699999999999,"5min":333.87400000000002,"15min":333.87400000000002},"last":15.957000000000001}]},{"osd":5,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":11.044,"5min":37.603999999999999,"15min":30.728000000000002},"min":{"1min":6.617,"5min":6.4249999999999998,"15min":6.4249999999999998},"max":{"1min":25.687999999999999,"5min":387.58100000000002,"15min":543.84100000000001},"last":9.4730000000000008},{"interface":"front","average":{"1min":23.571999999999999,"5min":32.238999999999997,"15min":23.466999999999999},"min":{"1min":11.239000000000001,"5min":6.3529999999999998,"15min":5.452},"max":{"1min":42.279000000000003,"5min":266.03800000000001,"15min":335.589},"last":18.02}]},{"osd":6,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":16.440000000000001,"5min":32.234000000000002,"15min":25.504999999999999},"min":{"1min":6.7729999999999997,"5min":6.1059999999999999,"15min":6.1059999999999999},"max":{"1min":30.492000000000001,"5min":311.358,"15min":370.30200000000002},"last":11.111000000000001},{"interface":"front","average":{"1min":19.751000000000001,"5min":33.356000000000002,"15min":26.114999999999998},"min":{"1min":11.441000000000001,"5min":8.2769999999999992,"15min":7.2389999999999999},"max":{"1min":40.555,"5min":230.27699999999999,"15min":230.27699999999999},"last":15.234999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:56:29 2024","interfaces":[{"interface":"back","average":{"1min":20.992000000000001,"5min":30.393999999999998,"15min":25.266999999999999},"min":{"1min":11.186,"5min":9.1110000000000007,"15min":8.8960000000000008},"max":{"1min":44.409999999999997,"5min":131.65000000000001,"15min":214.071},"last":14.638999999999999},{"interface":"front","average":{"1min":23.757999999999999,"5min":38.738999999999997,"15min":28.484999999999999},"min":{"1min":9.0990000000000002,"5min":8.8070000000000004,"15min":8.8070000000000004},"max":{"1min":53.152000000000001,"5min":348.38600000000002,"15min":348.38600000000002},"last":18.965}]}]},{"osd":6,"up_from":18,"seq":77309411776,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72252,"kb_used_data":4980,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299588,"statfs":{"total":96636764160,"available":96562778112,"internally_reserved":0,"allocated":5099520,"data_stored":2332603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":16.803000000000001,"5min":22.076000000000001,"15min":21.289999999999999},"min":{"1min":5.3380000000000001,"5min":5.3380000000000001,"15min":4.391},"max":{"1min":98.180000000000007,"5min":284.42500000000001,"15min":438.98899999999998},"last":9.3100000000000005},{"interface":"front","average":{"1min":28.359999999999999,"5min":27.478999999999999,"15min":21.431000000000001},"min":{"1min":8.7149999999999999,"5min":5.2160000000000002,"15min":4.8840000000000003},"max":{"1min":144.92400000000001,"5min":380.07600000000002,"15min":380.07600000000002},"last":10.728}]},{"osd":1,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":19.898,"5min":31.847999999999999,"15min":24.283000000000001},"min":{"1min":8.7289999999999992,"5min":6.883,"15min":4.7389999999999999},"max":{"1min":54.722999999999999,"5min":526.48099999999999,"15min":526.48099999999999},"last":13.352},{"interface":"front","average":{"1min":14.859999999999999,"5min":26.492000000000001,"15min":22.867999999999999},"min":{"1min":6.3600000000000003,"5min":4.3120000000000003,"15min":4.3120000000000003},"max":{"1min":44.970999999999997,"5min":395.72399999999999,"15min":395.72399999999999},"last":8.6229999999999993}]},{"osd":2,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":12.648,"5min":21.052,"15min":20.300999999999998},"min":{"1min":6.0350000000000001,"5min":5.8920000000000003,"15min":5.8920000000000003},"max":{"1min":35.268999999999998,"5min":209.78,"15min":213.83199999999999},"last":11.073},{"interface":"front","average":{"1min":22.007000000000001,"5min":31.407,"15min":26.654},"min":{"1min":6.7210000000000001,"5min":6.7210000000000001,"15min":6.7210000000000001},"max":{"1min":77.159000000000006,"5min":444.822,"15min":444.822},"last":19.274999999999999}]},{"osd":3,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":18.542999999999999,"5min":33.561,"15min":26.021000000000001},"min":{"1min":8.1959999999999997,"5min":6.1890000000000001,"15min":6.1890000000000001},"max":{"1min":59.645000000000003,"5min":527.40899999999999,"15min":527.40899999999999},"last":14.199},{"interface":"front","average":{"1min":17.805,"5min":25.48,"15min":23.213999999999999},"min":{"1min":6.4089999999999998,"5min":6.4050000000000002,"15min":6.4050000000000002},"max":{"1min":57.729999999999997,"5min":370.10000000000002,"15min":370.10000000000002},"last":17.207999999999998}]},{"osd":4,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":21.780000000000001,"5min":31.358000000000001,"15min":24.521999999999998},"min":{"1min":6.9160000000000004,"5min":6.3120000000000003,"15min":6.3120000000000003},"max":{"1min":95.962000000000003,"5min":385.49200000000002,"15min":385.49200000000002},"last":15.116},{"interface":"front","average":{"1min":21.581,"5min":26.248000000000001,"15min":21.954000000000001},"min":{"1min":8.0660000000000007,"5min":7.2130000000000001,"15min":6.7539999999999996},"max":{"1min":89.241,"5min":241.59200000000001,"15min":241.59200000000001},"last":17.986999999999998}]},{"osd":5,"last update":"Sun Jan 21 03:56:31 2024","interfaces":[{"interface":"back","average":{"1min":22.273,"5min":28.437000000000001,"15min":25.361999999999998},"min":{"1min":8.9830000000000005,"5min":8.2810000000000006,"15min":7.9450000000000003},"max":{"1min":98.718000000000004,"5min":488.01299999999998,"15min":488.01299999999998},"last":21.734000000000002},{"interface":"front","average":{"1min":21.077000000000002,"5min":23.109000000000002,"15min":25.189},"min":{"1min":9.093,"5min":8.1890000000000001,"15min":8.1890000000000001},"max":{"1min":101.967,"5min":197.38,"15min":333.43900000000002},"last":18.638000000000002}]},{"osd":7,"last update":"Sun Jan 21 03:56:06 2024","interfaces":[{"interface":"back","average":{"1min":23.504000000000001,"5min":30.591999999999999,"15min":25.545000000000002},"min":{"1min":10.676,"5min":7.1340000000000003,"15min":7.1340000000000003},"max":{"1min":100.84399999999999,"5min":322.54599999999999,"15min":322.54599999999999},"last":15.787000000000001},{"interface":"front","average":{"1min":24.988,"5min":29.010999999999999,"15min":25.050000000000001},"min":{"1min":13.025,"5min":9.6940000000000008,"15min":8.6699999999999999},"max":{"1min":90.594999999999999,"5min":218.30199999999999,"15min":218.30199999999999},"last":19.957000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411779,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72640,"kb_used_data":5368,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299200,"statfs":{"total":96636764160,"available":96562380800,"internally_reserved":0,"allocated":5496832,"data_stored":2730438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":9.1660000000000004,"5min":22.483000000000001,"15min":20.535},"min":{"1min":4.9420000000000002,"5min":4.6539999999999999,"15min":4.0940000000000003},"max":{"1min":28.151,"5min":241.49000000000001,"15min":346.541},"last":6.7619999999999996},{"interface":"front","average":{"1min":13.221,"5min":26.934000000000001,"15min":26.605},"min":{"1min":7.6550000000000002,"5min":6.1500000000000004,"15min":5.8819999999999997},"max":{"1min":24.053000000000001,"5min":236.81100000000001,"15min":524.904},"last":11.381}]},{"osd":1,"last update":"Sun Jan 21 03:56:24 2024","interfaces":[{"interface":"back","average":{"1min":12.57,"5min":24.992999999999999,"15min":19.875},"min":{"1min":7.0039999999999996,"5min":5.1429999999999998,"15min":4.907},"max":{"1min":22.702999999999999,"5min":284.07400000000001,"15min":368.12400000000002},"last":10.422000000000001},{"interface":"front","average":{"1min":19.931000000000001,"5min":33.564999999999998,"15min":26.795000000000002},"min":{"1min":8.7680000000000007,"5min":8.7680000000000007,"15min":7.5359999999999996},"max":{"1min":42.582999999999998,"5min":296.56400000000002,"15min":368.86099999999999},"last":16.978000000000002}]},{"osd":2,"last update":"Sun Jan 21 03:56:37 2024","interfaces":[{"interface":"back","average":{"1min":10.505000000000001,"5min":37.585999999999999,"15min":27.082000000000001},"min":{"1min":6.7990000000000004,"5min":6.6310000000000002,"15min":6.0579999999999998},"max":{"1min":21.158999999999999,"5min":2166.2739999999999,"15min":2166.2739999999999},"last":10.738},{"interface":"front","average":{"1min":19.701000000000001,"5min":45.871000000000002,"15min":35.814999999999998},"min":{"1min":12.010999999999999,"5min":7.1150000000000002,"15min":7.1150000000000002},"max":{"1min":50.802,"5min":2181.6190000000001,"15min":2181.6190000000001},"last":14.111000000000001}]},{"osd":3,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":16.638000000000002,"5min":30.282,"15min":26.837},"min":{"1min":6.8179999999999996,"5min":6.2770000000000001,"15min":6.2770000000000001},"max":{"1min":63.612000000000002,"5min":412.714,"15min":412.714},"last":12.765000000000001},{"interface":"front","average":{"1min":20.707999999999998,"5min":34.683999999999997,"15min":28.085999999999999},"min":{"1min":9.9399999999999995,"5min":9.6389999999999993,"15min":6.8840000000000003},"max":{"1min":57.048000000000002,"5min":314.28199999999998,"15min":314.28199999999998},"last":13.393000000000001}]},{"osd":4,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":21.478000000000002,"5min":39.764000000000003,"15min":27.739999999999998},"min":{"1min":12.226000000000001,"5min":7.7060000000000004,"15min":6.274},"max":{"1min":49.774999999999999,"5min":418.18299999999999,"15min":418.18299999999999},"last":17.875},{"interface":"front","average":{"1min":24.219000000000001,"5min":42.651000000000003,"15min":30.201000000000001},"min":{"1min":14.734999999999999,"5min":11.993,"15min":7.5490000000000004},"max":{"1min":57.712000000000003,"5min":387.125,"15min":387.125},"last":19.690999999999999}]},{"osd":6,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":17.492999999999999,"5min":32.762,"15min":25.588999999999999},"min":{"1min":8.6739999999999995,"5min":6.7050000000000001,"15min":6.7050000000000001},"max":{"1min":31.645,"5min":431.40899999999999,"15min":431.40899999999999},"last":14.757999999999999},{"interface":"front","average":{"1min":23.259,"5min":36.051000000000002,"15min":28.965},"min":{"1min":11.241,"5min":9.0809999999999995,"15min":8.5470000000000006},"max":{"1min":61.225999999999999,"5min":432.19,"15min":432.19},"last":15.372999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:56:00 2024","interfaces":[{"interface":"back","average":{"1min":23.452000000000002,"5min":38.652000000000001,"15min":29.713000000000001},"min":{"1min":13.459,"5min":9.7560000000000002,"15min":7.8639999999999999},"max":{"1min":48.789999999999999,"5min":281.06099999999998,"15min":281.06099999999998},"last":18.800000000000001},{"interface":"front","average":{"1min":12.581,"5min":26.823,"15min":22.974},"min":{"1min":6.5170000000000003,"5min":6.5170000000000003,"15min":6.5170000000000003},"max":{"1min":36.563000000000002,"5min":282.29000000000002,"15min":282.29000000000002},"last":9.5470000000000006}]}]},{"osd":7,"up_from":14,"seq":60129542592,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72252,"kb_used_data":4980,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299588,"statfs":{"total":96636764160,"available":96562778112,"internally_reserved":0,"allocated":5099520,"data_stored":2332603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:17 2024","interfaces":[{"interface":"back","average":{"1min":11.395,"5min":20.713000000000001,"15min":16.225999999999999},"min":{"1min":5.0099999999999998,"5min":4.7450000000000001,"15min":4.1929999999999996},"max":{"1min":27.079000000000001,"5min":239.37799999999999,"15min":304.20800000000003},"last":9.6270000000000007},{"interface":"front","average":{"1min":10.247999999999999,"5min":21.539000000000001,"15min":16.581},"min":{"1min":5.9039999999999999,"5min":5.2709999999999999,"15min":4.2679999999999998},"max":{"1min":31.059999999999999,"5min":181.04599999999999,"15min":181.04599999999999},"last":6.3230000000000004}]},{"osd":1,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":12.342000000000001,"5min":19.533000000000001,"15min":20.614000000000001},"min":{"1min":6.0869999999999997,"5min":5.6580000000000004,"15min":5.0419999999999998},"max":{"1min":27.402999999999999,"5min":138.82499999999999,"15min":248.17500000000001},"last":7.0720000000000001},{"interface":"front","average":{"1min":14.029,"5min":23.824999999999999,"15min":23.670999999999999},"min":{"1min":7.9109999999999996,"5min":6.3769999999999998,"15min":6.3769999999999998},"max":{"1min":26.454000000000001,"5min":214.55000000000001,"15min":277.85700000000003},"last":10.675000000000001}]},{"osd":2,"last update":"Sun Jan 21 03:56:17 2024","interfaces":[{"interface":"back","average":{"1min":17.594999999999999,"5min":36.784999999999997,"15min":27.683},"min":{"1min":7.2869999999999999,"5min":7.2290000000000001,"15min":6.1779999999999999},"max":{"1min":47.972999999999999,"5min":683.21000000000004,"15min":683.21000000000004},"last":11.401999999999999},{"interface":"front","average":{"1min":16.43,"5min":24.434999999999999,"15min":22.135999999999999},"min":{"1min":9.0399999999999991,"5min":6.3890000000000002,"15min":6.3890000000000002},"max":{"1min":35.869,"5min":150.369,"15min":367.30000000000001},"last":16.715}]},{"osd":3,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":17.163,"5min":27.100000000000001,"15min":22.632000000000001},"min":{"1min":10.734999999999999,"5min":7.1470000000000002,"15min":6.8040000000000003},"max":{"1min":24.895,"5min":192.14599999999999,"15min":192.14599999999999},"last":12.462999999999999},{"interface":"front","average":{"1min":16.158000000000001,"5min":24.597000000000001,"15min":21.649000000000001},"min":{"1min":7.851,"5min":7.8090000000000002,"15min":6.5960000000000001},"max":{"1min":29.548999999999999,"5min":166.846,"15min":210.006},"last":13.444000000000001}]},{"osd":4,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":20.638999999999999,"5min":27.516999999999999,"15min":22.658999999999999},"min":{"1min":9.4700000000000006,"5min":7.6260000000000003,"15min":7.2309999999999999},"max":{"1min":43.503,"5min":197.31,"15min":268.416},"last":15.742000000000001},{"interface":"front","average":{"1min":17.760999999999999,"5min":25.548999999999999,"15min":21.73},"min":{"1min":7.516,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":33.429000000000002,"5min":143.07400000000001,"15min":168.95599999999999},"last":18.532}]},{"osd":5,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":17.228999999999999,"5min":25.649999999999999,"15min":24.148},"min":{"1min":7.3029999999999999,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":25.670000000000002,"5min":149.56,"15min":226.947},"last":19.751000000000001},{"interface":"front","average":{"1min":17.081,"5min":24.050000000000001,"15min":23.798999999999999},"min":{"1min":8.6419999999999995,"5min":7.5839999999999996,"15min":7.5839999999999996},"max":{"1min":41.097000000000001,"5min":137.00200000000001,"15min":248.94800000000001},"last":14.753}]},{"osd":6,"last update":"Sun Jan 21 03:56:07 2024","interfaces":[{"interface":"back","average":{"1min":21.542000000000002,"5min":27.488,"15min":26.065999999999999},"min":{"1min":10.897,"5min":9.4459999999999997,"15min":9.4459999999999997},"max":{"1min":55.081000000000003,"5min":193.16,"15min":295.12099999999998},"last":17.716999999999999},{"interface":"front","average":{"1min":19.73,"5min":28.16,"15min":26.513999999999999},"min":{"1min":11.029,"5min":9.3219999999999992,"15min":9.3219999999999992},"max":{"1min":55.758000000000003,"5min":178.01900000000001,"15min":278.71899999999999},"last":21.026}]}]},{"osd":4,"up_from":14,"seq":60129542591,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72248,"kb_used_data":4976,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299592,"statfs":{"total":96636764160,"available":96562782208,"internally_reserved":0,"allocated":5095424,"data_stored":2332598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":8.7889999999999997,"5min":15.209,"15min":16.693000000000001},"min":{"1min":4.9939999999999998,"5min":4.4660000000000002,"15min":4.4100000000000001},"max":{"1min":18.459,"5min":220.72900000000001,"15min":220.72900000000001},"last":6.4820000000000002},{"interface":"front","average":{"1min":12.965,"5min":16.300000000000001,"15min":18.149999999999999},"min":{"1min":6.6360000000000001,"5min":4.6029999999999998,"15min":4.6029999999999998},"max":{"1min":45.616,"5min":101.768,"15min":204.142},"last":9.8209999999999997}]},{"osd":1,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":12.475,"5min":22.411999999999999,"15min":18.585000000000001},"min":{"1min":6.6440000000000001,"5min":4.9359999999999999,"15min":4.9359999999999999},"max":{"1min":26.292999999999999,"5min":272.61399999999998,"15min":272.61399999999998},"last":16.635000000000002},{"interface":"front","average":{"1min":16.969000000000001,"5min":24.193000000000001,"15min":20.829000000000001},"min":{"1min":9.1150000000000002,"5min":6.8719999999999999,"15min":5.6200000000000001},"max":{"1min":49.414999999999999,"5min":142.97300000000001,"15min":223.49299999999999},"last":12.356999999999999}]},{"osd":2,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":23.027000000000001,"5min":29.632000000000001,"15min":24.187999999999999},"min":{"1min":9.3979999999999997,"5min":7.4320000000000004,"15min":6.6879999999999997},"max":{"1min":77.858999999999995,"5min":285.61000000000001,"15min":285.61000000000001},"last":14.489000000000001},{"interface":"front","average":{"1min":21.07,"5min":26.555,"15min":22.710999999999999},"min":{"1min":7.6769999999999996,"5min":6.899,"15min":6.2080000000000002},"max":{"1min":101.676,"5min":298.892,"15min":298.892},"last":7.6680000000000001}]},{"osd":3,"last update":"Sun Jan 21 03:56:14 2024","interfaces":[{"interface":"back","average":{"1min":14.119,"5min":19.032,"15min":20.623999999999999},"min":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"max":{"1min":48.497,"5min":132.958,"15min":200.697},"last":11.619999999999999},{"interface":"front","average":{"1min":19.265000000000001,"5min":23.931999999999999,"15min":21.149000000000001},"min":{"1min":8.9890000000000008,"5min":7.5250000000000004,"15min":7.1539999999999999},"max":{"1min":60.786000000000001,"5min":166.66200000000001,"15min":188.06399999999999},"last":10.814}]},{"osd":5,"last update":"Sun Jan 21 03:55:58 2024","interfaces":[{"interface":"back","average":{"1min":17.315000000000001,"5min":26.736000000000001,"15min":24.969000000000001},"min":{"1min":10.885999999999999,"5min":8.2970000000000006,"15min":6.7359999999999998},"max":{"1min":30.718,"5min":182.53700000000001,"15min":470.87700000000001},"last":18.937999999999999},{"interface":"front","average":{"1min":13.831,"5min":23.012,"15min":24.087},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":6.944},"max":{"1min":23.405000000000001,"5min":145.125,"15min":471.56999999999999},"last":17.356000000000002}]},{"osd":6,"last update":"Sun Jan 21 03:56:40 2024","interfaces":[{"interface":"back","average":{"1min":19.273,"5min":26.210999999999999,"15min":21.297999999999998},"min":{"1min":8.6690000000000005,"5min":8.6690000000000005,"15min":6.9290000000000003},"max":{"1min":61.551000000000002,"5min":208.42099999999999,"15min":285.87},"last":13.003},{"interface":"front","average":{"1min":15.960000000000001,"5min":25.577999999999999,"15min":23.658999999999999},"min":{"1min":9.7910000000000004,"5min":9.5129999999999999,"15min":7.5430000000000001},"max":{"1min":25.187999999999999,"5min":252.47999999999999,"15min":252.47999999999999},"last":18.138999999999999}]},{"osd":7,"last update":"Sun Jan 21 03:55:58 2024","interfaces":[{"interface":"back","average":{"1min":20.036999999999999,"5min":25.931000000000001,"15min":22.280000000000001},"min":{"1min":12.244,"5min":9.3040000000000003,"15min":7.1820000000000004},"max":{"1min":44.164999999999999,"5min":141.51300000000001,"15min":229.04900000000001},"last":15.176},{"interface":"front","average":{"1min":22.382999999999999,"5min":28.206,"15min":22.698},"min":{"1min":15.759,"5min":12.417999999999999,"15min":7.4299999999999997},"max":{"1min":42.031999999999996,"5min":179.12799999999999,"15min":179.12799999999999},"last":15.888}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T03:56:57.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-21T03:56:57.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.165+0000 7fb869b89640 10 monclient: get_monmap_and_config 2024-01-21T03:56:57.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.165+0000 7fb869b89640 10 monclient: build_initial_monmap 2024-01-21T03:56:57.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.165+0000 7fb869b89640 10 monclient: monmap: 2024-01-21T03:56:57.166 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:57.166 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:57.166 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:56:57.166620+0000 2024-01-21T03:56:57.166 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:56:57.166620+0000 2024-01-21T03:56:57.166 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:57.167 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:57.167 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-01-21T03:56:57.167 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-01-21T03:56:57.167 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-01-21T03:56:57.167 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-01-21T03:56:57.167 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-01-21T03:56:57.167 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:57.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.166+0000 7fb869b89640 1 Processor -- start 2024-01-21T03:56:57.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.166+0000 7fb869b89640 1 -- start start 2024-01-21T03:56:57.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.166+0000 7fb869b89640 10 monclient: init 2024-01-21T03:56:57.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.166+0000 7fb869b89640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:57.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.166+0000 7fb869b89640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-01-21T03:56:57.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.167+0000 7fb869b89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb8640a8b20 addr v1:172.21.15.177:6789/0 2024-01-21T03:56:57.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.167+0000 7fb869b89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb8640a9d10 addr v1:172.21.15.7:6789/0 2024-01-21T03:56:57.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.167+0000 7fb869b89640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640b2310 0x7fb86413f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:57.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.167+0000 7fb869b89640 10 monclient(hunting): picked mon.noname-b con 0x7fb8640b2310 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:57.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.167+0000 7fb869b89640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8641456f0 con 0x7fb8640a9d10 2024-01-21T03:56:57.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.167+0000 7fb869b89640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8640a98c0 con 0x7fb8640a8b20 2024-01-21T03:56:57.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.167+0000 7fb869b89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.167+0000 7fb869b89640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb864146290 con 0x7fb8640b2310 2024-01-21T03:56:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.167+0000 7fb869b89640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.167+0000 7fb869b89640 10 monclient(hunting): authenticate will time out at 3563.482839s 2024-01-21T03:56:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.167+0000 7fb863fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640b2310 0x7fb86413f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.167+0000 7fb8637fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fb8640a8b20 0x7fb864145b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.7:49122/0 (socket says 172.21.15.7:49122) 2024-01-21T03:56:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.167+0000 7fb8637fe640 1 -- 172.21.15.7:0/2446622715 learned_addr learned my addr 172.21.15.7:0/2446622715 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-01-21T03:56:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.168+0000 7fb863fff640 10 monclient(hunting): get_auth_request con 0x7fb8640b2310 auth_method 0 2024-01-21T03:56:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.168+0000 7fb863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.168+0000 7fb863fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.168+0000 7fb863fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.168+0000 7fb863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.168+0000 7fb863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.168+0000 7fb863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:57.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.169+0000 7fb8627fc640 1 -- 172.21.15.7:0/2446622715 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821638973 0 0) 0x7fb8640a98c0 con 0x7fb8640a8b20 2024-01-21T03:56:57.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb8627fc640 10 monclient(hunting): _negotiate 2024-01-21T03:56:57.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb8627fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:57.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb8627fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:57.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb8627fc640 10 monclient(hunting): my global_id is 23251 2024-01-21T03:56:57.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb8627fc640 1 -- 172.21.15.7:0/2446622715 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb84c003520 con 0x7fb8640a8b20 2024-01-21T03:56:57.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb863fff640 10 monclient(hunting): handle_auth_done global_id 18089 payload 274 2024-01-21T03:56:57.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb863fff640 1 -- 172.21.15.7:0/2446622715 >> v1:172.21.15.177:6789/0 conn(0x7fb8640a8b20 legacy=0x7fb864145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:57.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb863fff640 1 -- 172.21.15.7:0/2446622715 >> v1:172.21.15.7:6789/0 conn(0x7fb8640a9d10 legacy=0x7fb86413f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:57.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb8627fc640 1 -- 172.21.15.7:0/2446622715 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264985184 0 0) 0x7fb8641456f0 con 0x7fb8640a9d10 2024-01-21T03:56:57.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb863fff640 10 monclient: _finish_hunting 0 2024-01-21T03:56:57.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb863fff640 1 monclient: found mon.noname-b 2024-01-21T03:56:57.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb863fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:57.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb863fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-01-21T03:56:57.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb863fff640 1 -- 172.21.15.7:0/2446622715 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8641463d0 con 0x7fb8640b2310 2024-01-21T03:56:57.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.170+0000 7fb8627fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-21T03:56:57.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.172+0000 7fb863fff640 1 --2- 172.21.15.7:0/2446622715 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640b2310 0x7fb86413f790 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7fb85c009dc0 tx=0x7fb85c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6590e9168360681 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:57.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb8627fc640 1 -- 172.21.15.7:0/2446622715 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c02cc40 con 0x7fb8640b2310 2024-01-21T03:56:57.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb8627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:57.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb8627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb8627fc640 10 monclient: dump: 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb8627fc640 10 monclient: _finish_auth 0 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb8627fc640 10 monclient: _check_auth_tickets 2024-01-21T03:56:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb8627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb8627fc640 1 -- 172.21.15.7:0/2446622715 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c02cda0 con 0x7fb8640b2310 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb8627fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb8627fc640 1 -- 172.21.15.7:0/2446622715 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c034660 con 0x7fb8640b2310 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb869b89640 5 monclient: authenticate success, global_id 18089 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb869b89640 10 monclient: get_monmap_and_config success 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb869b89640 4 set_mon_vals no callback set 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb8627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb8627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.176+0000 7fb8627fc640 10 monclient: dump: 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:57.179 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:57.180 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:57.180 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:57.180 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:57.180 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:57.180 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:57.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.177+0000 7fb869b89640 10 monclient: shutdown 2024-01-21T03:56:57.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.177+0000 7fb869b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:57.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.177+0000 7fb869b89640 1 -- 172.21.15.7:0/2446622715 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640b2310 msgr2=0x7fb86413f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:57.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.177+0000 7fb869b89640 1 --2- 172.21.15.7:0/2446622715 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640b2310 0x7fb86413f790 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7fb85c009dc0 tx=0x7fb85c02be90 comp rx=0 tx=0).stop 2024-01-21T03:56:57.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.177+0000 7fb869b89640 1 -- 172.21.15.7:0/2446622715 shutdown_connections 2024-01-21T03:56:57.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.177+0000 7fb869b89640 1 --2- 172.21.15.7:0/2446622715 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640b2310 0x7fb86413f790 unknown :-1 s=CLOSED pgs=3163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.177+0000 7fb869b89640 1 -- 172.21.15.7:0/2446622715 >> 172.21.15.7:0/2446622715 conn(0x7fb864066fb0 msgr2=0x7fb8640b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.178+0000 7fb869b89640 1 -- 172.21.15.7:0/2446622715 shutdown_connections 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.178+0000 7fb869b89640 1 -- 172.21.15.7:0/2446622715 wait complete. 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.178+0000 7fb869b89640 10 monclient: build_initial_monmap 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.178+0000 7fb869b89640 10 monclient: monmap: 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-01-21T03:56:57.184 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.178+0000 7fb869b89640 1 librados: starting msgr at 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.178+0000 7fb869b89640 1 librados: starting objecter 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 1 Processor -- start 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 1 -- start start 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 1 librados: setting wanted keys 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 1 librados: calling monclient init 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 10 monclient: init 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 10 monclient: _reopen_session rank -1 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640a8b20 0x7fb864143020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 10 monclient(hunting): picked mon.noname-a con 0x7fb8640a8b20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8640a9d10 0x7fb864138900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 10 monclient(hunting): picked mon.noname-b con 0x7fb8640a9d10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-01-21T03:56:57.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640b2310 0x7fb864136d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 10 monclient(hunting): picked mon.noname-c con 0x7fb8640b2310 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864143560 con 0x7fb8640a8b20 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864138200 con 0x7fb8640a9d10 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 10 monclient(hunting): start opening mon connection 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8641384e0 con 0x7fb8640b2310 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 10 monclient(hunting): _renew_subs 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.179+0000 7fb869b89640 10 monclient(hunting): authenticate will time out at 3563.495616s 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.180+0000 7fb863fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640b2310 0x7fb864136d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.180+0000 7fb862ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8640a9d10 0x7fb864138900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.180+0000 7fb8637fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640a8b20 0x7fb864143020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.180+0000 7fb862ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8640a9d10 0x7fb864138900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.7:34138/0 (socket says 172.21.15.7:34138) 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.180+0000 7fb862ffd640 1 -- 172.21.15.7:0/1756335579 learned_addr learned my addr 172.21.15.7:0/1756335579 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.180+0000 7fb862ffd640 10 monclient(hunting): get_auth_request con 0x7fb8640a9d10 auth_method 0 2024-01-21T03:56:57.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.180+0000 7fb862ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.180+0000 7fb862ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.180+0000 7fb862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.180+0000 7fb863fff640 10 monclient(hunting): get_auth_request con 0x7fb8640b2310 auth_method 0 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.180+0000 7fb863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.180+0000 7fb863fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.180+0000 7fb863fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.181+0000 7fb8637fe640 10 monclient(hunting): get_auth_request con 0x7fb8640a8b20 auth_method 0 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.181+0000 7fb8637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.181+0000 7fb8637fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.181+0000 7fb8637fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.181+0000 7fb862ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.181+0000 7fb862ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.181+0000 7fb862ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.181+0000 7fb863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.181+0000 7fb863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:57.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.181+0000 7fb863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:57.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.181+0000 7fb8637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T03:56:57.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.181+0000 7fb8637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T03:56:57.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.181+0000 7fb8637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T03:56:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.183+0000 7fb862ffd640 10 monclient(hunting): handle_auth_done global_id 23254 payload 824 2024-01-21T03:56:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.183+0000 7fb862ffd640 1 -- 172.21.15.7:0/1756335579 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640b2310 msgr2=0x7fb864136d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.183+0000 7fb862ffd640 1 --2- 172.21.15.7:0/1756335579 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640b2310 0x7fb864136d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.183+0000 7fb862ffd640 1 -- 172.21.15.7:0/1756335579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640a8b20 msgr2=0x7fb864143020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.183+0000 7fb862ffd640 1 --2- 172.21.15.7:0/1756335579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640a8b20 0x7fb864143020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.183+0000 7fb862ffd640 10 monclient: _finish_hunting 0 2024-01-21T03:56:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.183+0000 7fb862ffd640 1 monclient: found mon.noname-b 2024-01-21T03:56:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.183+0000 7fb862ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T03:56:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.183+0000 7fb862ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-01-21T03:56:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.183+0000 7fb862ffd640 1 -- 172.21.15.7:0/1756335579 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c009a70 con 0x7fb8640a9d10 2024-01-21T03:56:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.186+0000 7fb862ffd640 1 --2- 172.21.15.7:0/1756335579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8640a9d10 0x7fb864138900 secure :-1 s=READY pgs=5836 cs=0 l=1 rev1=1 crypto rx=0x7fb85800c550 tx=0x7fb85800ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.186+0000 7fb84bfff640 10 client.?.objecter ms_handle_connect 0x7fb8640a9d10 2024-01-21T03:56:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.186+0000 7fb84bfff640 10 client.?.objecter resend_mon_ops 2024-01-21T03:56:57.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb84bfff640 1 -- 172.21.15.7:0/1756335579 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8580035f0 con 0x7fb8640a9d10 2024-01-21T03:56:57.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb84bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:57.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb84bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T03:56:57.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb84bfff640 10 monclient: dump: 2024-01-21T03:56:57.190 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:57.190 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb84bfff640 10 monclient: _finish_auth 0 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb84bfff640 10 monclient: _check_auth_tickets 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb84bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb84bfff640 1 -- 172.21.15.7:0/1756335579 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb858003750 con 0x7fb8640a9d10 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb84bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb84bfff640 1 -- 172.21.15.7:0/1756335579 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb858003b00 con 0x7fb8640a9d10 2024-01-21T03:56:57.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb84bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb84bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb84bfff640 10 monclient: dump: 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 8de9e7dd-e982-4048-83dd-ee52350dd56d 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-01-21T03:18:14.727370+0000 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb869b89640 5 monclient: authenticate success, global_id 23254 2024-01-21T03:56:57.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb869b89640 10 monclient: _renew_subs 2024-01-21T03:56:57.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb869b89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:57.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb869b89640 1 -- 172.21.15.7:0/1756335579 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8641448e0 con 0x7fb8640a9d10 2024-01-21T03:56:57.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb861ffb640 4 set_mon_vals no callback set 2024-01-21T03:56:57.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb869b89640 10 client.23254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T03:56:57.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb869b89640 10 monclient: _renew_subs 2024-01-21T03:56:57.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb869b89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:57.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb869b89640 1 -- 172.21.15.7:0/1756335579 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb864138e40 con 0x7fb8640a9d10 2024-01-21T03:56:57.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.189+0000 7fb869b89640 1 librados: init done 2024-01-21T03:56:57.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.190+0000 7fb869b89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:57.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.190+0000 7fb869b89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:57.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.190+0000 7fb869b89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:57.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.190+0000 7fb869b89640 1 -- 172.21.15.7:0/1756335579 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8641366f0 con 0x7fb8640a9d10 2024-01-21T03:56:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.432+0000 7fb84bfff640 1 -- 172.21.15.7:0/1756335579 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb85801b3b0 con 0x7fb8640a9d10 2024-01-21T03:56:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.432+0000 7fb84bfff640 1 --2- 172.21.15.7:0/1756335579 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb85006c8b0 0x7fb85006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T03:56:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.432+0000 7fb8637fe640 1 --2- 172.21.15.7:0/1756335579 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb85006c8b0 0x7fb85006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T03:56:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.432+0000 7fb8637fe640 10 monclient: get_auth_request con 0x7fb85006c8b0 auth_method 0 2024-01-21T03:56:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.433+0000 7fb84bfff640 10 client.23254.objecter ms_handle_connect 0x7fb85006c8b0 2024-01-21T03:56:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.433+0000 7fb8637fe640 1 --2- 172.21.15.7:0/1756335579 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb85006c8b0 0x7fb85006ed70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb8540011e0 tx=0x7fb854007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T03:56:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.434+0000 7fb84bfff640 1 -- 172.21.15.7:0/1756335579 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1294..1294 src has 999..1294) v4 ==== 6387+0+0 (secure 0 0 0) 0x7fb858057cc0 con 0x7fb8640a9d10 2024-01-21T03:56:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.434+0000 7fb84bfff640 10 client.23254.objecter ms_dispatch 0x7fb864000b90 osd_map(1294..1294 src has 999..1294) v4 2024-01-21T03:56:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.434+0000 7fb84bfff640 3 client.23254.objecter handle_osd_map got epochs [1294,1294] > 0 2024-01-21T03:56:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.434+0000 7fb84bfff640 3 client.23254.objecter handle_osd_map decoding full epoch 1294 2024-01-21T03:56:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.434+0000 7fb84bfff640 20 client.23254.objecter dump_active .. 0 homeless 2024-01-21T03:56:57.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.444+0000 7fb84bfff640 1 -- 172.21.15.7:0/1756335579 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb858054000 con 0x7fb8640a9d10 2024-01-21T03:56:57.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.444+0000 7fb84bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T03:56:57.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.444+0000 7fb84bfff640 10 monclient: _finish_command 1 = system:0 2024-01-21T03:56:57.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.625+0000 7fb869b89640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-01-21T03:56:57.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.625+0000 7fb869b89640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-01-21T03:56:57.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.625+0000 7fb869b89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-01-21T03:56:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.625+0000 7fb869b89640 1 -- 172.21.15.7:0/1756335579 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fb8640aa7b0 con 0x7fb8640a9d10 2024-01-21T03:56:57.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.754+0000 7fb84bfff640 1 -- 172.21.15.7:0/1756335579 <== mon.1 v2:172.21.15.177:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fb858054000 con 0x7fb8640a9d10 2024-01-21T03:56:57.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.754+0000 7fb84bfff640 10 monclient: handle_config config(1 keys) v1 2024-01-21T03:56:57.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.754+0000 7fb861ffb640 4 set_mon_vals no callback set 2024-01-21T03:56:57.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.754+0000 7fb861ffb640 10 set_mon_vals mon_health_to_clog = false 2024-01-21T03:56:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.760+0000 7fb84bfff640 1 -- 172.21.15.7:0/1756335579 <== mon.1 v2:172.21.15.177:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v8)=0 v8) v1 ==== 123+0+0 (secure 0 0 0) 0x7fb8580170c0 con 0x7fb8640a9d10 2024-01-21T03:56:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.760+0000 7fb84bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-01-21T03:56:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.760+0000 7fb84bfff640 10 monclient: _finish_command 2 = system:0 2024-01-21T03:56:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 10 librados: watch_flush enter 2024-01-21T03:56:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 10 librados: watch_flush exit 2024-01-21T03:56:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 20 client.23254.objecter shutdown clearing up homeless session... 2024-01-21T03:56:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 10 client.23254.objecter successfully canceled tick 2024-01-21T03:56:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 1 -- 172.21.15.7:0/1756335579 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb85006c8b0 msgr2=0x7fb85006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 1 --2- 172.21.15.7:0/1756335579 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb85006c8b0 0x7fb85006ed70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb8540011e0 tx=0x7fb854007d30 comp rx=0 tx=0).stop 2024-01-21T03:56:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 10 monclient: shutdown 2024-01-21T03:56:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T03:56:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 1 -- 172.21.15.7:0/1756335579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8640a9d10 msgr2=0x7fb864138900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T03:56:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 1 --2- 172.21.15.7:0/1756335579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8640a9d10 0x7fb864138900 secure :-1 s=READY pgs=5836 cs=0 l=1 rev1=1 crypto rx=0x7fb85800c550 tx=0x7fb85800ca10 comp rx=0 tx=0).stop 2024-01-21T03:56:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 1 -- 172.21.15.7:0/1756335579 shutdown_connections 2024-01-21T03:56:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 1 --2- 172.21.15.7:0/1756335579 >> [v2:172.21.15.7:6800/2590781548,v1:172.21.15.7:6801/2590781548] conn(0x7fb85006c8b0 0x7fb85006ed70 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 1 --2- 172.21.15.7:0/1756335579 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb8640b2310 0x7fb864136d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 1 --2- 172.21.15.7:0/1756335579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8640a9d10 0x7fb864138900 unknown :-1 s=CLOSED pgs=5836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 1 --2- 172.21.15.7:0/1756335579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8640a8b20 0x7fb864143020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T03:56:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 1 -- 172.21.15.7:0/1756335579 >> 172.21.15.7:0/1756335579 conn(0x7fb864066fb0 msgr2=0x7fb864134d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T03:56:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 1 -- 172.21.15.7:0/1756335579 shutdown_connections 2024-01-21T03:56:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 1 -- 172.21.15.7:0/1756335579 wait complete. 2024-01-21T03:56:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-01-21T03:56:57.762+0000 7fb869b89640 1 librados: shutdown 2024-01-21T03:56:57.774 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-21T03:56:57.774 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-21T03:56:57.774 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-21T03:56:57.774 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T03:56:57.787 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:56:57.778+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 34470) UID: 0 2024-01-21T03:56:57.788 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-01-21T03:56:57.779+0000 b47e640 -1 osd.0 1294 *** Got signal Terminated *** 2024-01-21T03:57:03.876 INFO:tasks.ceph.osd.0:Stopped 2024-01-21T03:57:03.876 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-21T03:57:03.876 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T03:57:03.891 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:57:03.880+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 34464) UID: 0 2024-01-21T03:57:03.893 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-01-21T03:57:03.882+0000 b47e640 -1 osd.1 1296 *** Got signal Terminated *** 2024-01-21T03:57:03.917 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-01-21T03:57:06.502 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:20.817 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:06.507 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:20.821 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:06.529 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:20.843 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:06.554 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:20.813 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:06.557 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:20.816 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:06.560 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:20.875 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:06.562 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:20.877 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:06.576 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:20.835 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:06.584 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:20.898 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:10.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-01-21T03:57:11.517 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:25.832 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:11.521 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:25.836 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:11.538 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:25.852 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:11.569 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:25.828 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:11.571 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:25.831 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:11.575 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:25.889 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:11.578 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:25.892 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:11.584 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:25.844 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:11.591 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:25.905 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:15.778 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:30.092 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:15.829 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:30.143 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:15.843 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:30.157 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:15.876 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:30.135 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:15.879 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:30.139 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:15.885 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:30.199 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:15.886 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:30.201 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:15.893 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:30.152 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:15.901 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:30.215 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:26.563 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:40.877 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:26.568 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:40.882 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:26.587 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:40.901 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:26.616 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:40.875 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:26.619 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:40.878 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:26.622 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:40.936 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:26.625 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:40.939 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:26.633 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:40.892 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:26.644 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:40.959 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:31.578 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:45.892 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:31.582 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:45.897 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:31.596 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:38:45.910 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:31.631 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:45.890 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:31.635 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:45.895 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:31.636 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:45.950 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:31.637 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:45.951 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:31.647 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:38:45.906 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:31.651 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:38:45.965 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:46.617 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:00.931 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:46.621 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:00.935 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:46.643 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:00.957 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:46.670 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:00.929 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:46.671 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:00.985 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:46.674 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:00.933 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:46.676 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:00.991 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:46.690 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:00.949 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:46.692 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:01.007 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:51.632 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:05.946 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:51.633 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:05.948 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:51.649 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:05.963 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:51.677 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:05.936 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:51.679 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:05.939 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:51.683 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:05.998 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:51.686 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:06.000 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:51.692 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:05.951 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:57:51.701 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:06.016 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:06.678 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:20.991 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:06.680 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:20.994 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:06.697 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:21.011 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:06.731 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:21.045 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:06.732 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:20.991 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:06.733 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:20.992 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:06.734 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:21.048 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:06.747 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:21.006 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:06.753 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:21.067 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:11.648 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:25.962 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:11.699 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:26.013 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:11.699 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:25.958 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:11.701 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:25.960 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:11.715 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:25.975 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:11.719 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:26.033 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:11.759 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:26.073 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:11.764 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:26.078 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:11.777 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:26.091 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:26.731 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:41.045 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:26.735 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:41.049 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:26.752 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:41.066 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:26.785 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:41.044 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:26.788 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:41.048 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:26.789 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:41.103 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:26.791 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:41.105 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:26.805 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:41.064 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:26.808 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:41.122 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:29.298 INFO:tasks.ceph.osd.1:Stopped 2024-01-21T03:58:29.298 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-21T03:58:29.299 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T03:58:29.311 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:58:29.302+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 34477) UID: 0 2024-01-21T03:58:29.311 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-01-21T03:58:29.303+0000 b47e640 -1 osd.2 1298 *** Got signal Terminated *** 2024-01-21T03:58:31.745 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:46.059 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:31.751 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:46.065 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:31.772 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:39:46.086 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:31.794 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:46.053 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:31.798 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:46.057 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:31.805 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:46.119 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:31.808 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:46.122 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:31.811 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:39:46.070 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:31.823 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:39:46.137 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:46.819 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:01.133 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:46.825 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:01.140 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:46.846 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:01.160 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:46.879 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:01.138 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:46.881 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:01.140 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:46.896 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:01.210 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:46.897 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:01.156 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:46.898 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:01.212 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:46.917 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:01.231 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:51.833 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:06.147 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:51.839 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:06.153 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:51.853 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:06.167 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:51.885 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:06.144 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:51.888 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:06.147 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:51.900 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:06.159 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:51.909 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:06.223 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:51.912 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:06.226 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:58:51.926 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:06.239 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:06.892 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:21.206 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:06.895 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:21.209 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:06.913 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:21.227 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:06.946 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:21.204 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:06.947 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:21.261 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:06.947 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:21.206 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:06.950 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:21.264 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:06.963 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:21.222 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:06.967 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:21.281 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:11.905 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:26.219 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:11.911 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:26.225 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:11.924 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:26.238 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:11.960 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:26.273 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:11.961 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:26.220 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:11.961 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:26.275 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:11.963 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:26.221 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:11.975 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:26.289 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:11.975 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:26.234 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:12.009 INFO:tasks.ceph.osd.2:Stopped 2024-01-21T03:59:12.009 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-21T03:59:12.009 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T03:59:12.022 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:59:12.012+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 34478) UID: 0 2024-01-21T03:59:12.022 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-01-21T03:59:12.013+0000 b47e640 -1 osd.3 1302 *** Got signal Terminated *** 2024-01-21T03:59:18.110 INFO:tasks.ceph.osd.3:Stopped 2024-01-21T03:59:18.111 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-21T03:59:18.111 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T03:59:18.124 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:59:18.114+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 34516) UID: 0 2024-01-21T03:59:18.127 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-01-21T03:59:18.115+0000 b47e640 -1 osd.4 1304 *** Got signal Terminated *** 2024-01-21T03:59:22.028 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:36.342 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:22.031 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:36.345 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:22.048 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:36.362 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:22.080 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:36.339 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:22.081 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:36.394 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:22.082 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:36.341 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:22.082 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:36.396 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:22.099 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:36.357 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:22.100 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:36.414 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:32.026 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:46.340 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:32.030 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:46.344 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:32.069 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:46.382 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:32.073 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:46.332 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:32.076 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:46.335 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:32.106 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:46.420 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:32.107 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:46.421 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:32.114 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:46.373 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:32.120 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:46.434 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:37.007 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:51.321 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:37.009 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:51.323 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:37.025 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:40:51.338 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:37.053 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:51.312 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:37.056 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:51.315 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:37.057 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:51.370 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:37.062 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:51.376 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:37.070 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:40:51.329 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:37.077 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:40:51.390 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:52.049 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:06.362 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:52.054 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:06.368 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:52.070 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:06.384 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:52.102 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:06.361 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:52.105 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:06.419 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:52.105 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:06.364 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:52.108 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:06.422 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:52.120 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:06.379 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:52.123 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:06.437 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:57.066 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:11.379 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:57.070 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:11.384 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:57.083 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:11.397 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:57.118 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:11.377 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:57.119 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:11.378 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:57.124 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:11.437 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:57.127 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:11.441 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:57.129 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:11.388 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T03:59:57.139 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:11.453 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:12.112 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:26.425 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:12.119 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:26.432 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:12.134 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:26.448 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:12.161 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:26.420 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:12.164 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:26.423 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:12.167 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:26.481 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:12.169 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:26.483 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:12.178 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:26.437 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:12.185 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:26.498 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.149 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:31.463 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.151 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:31.464 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.165 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:31.478 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.188 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:31.447 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.192 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:31.451 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.200 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:31.514 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.203 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:31.461 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.204 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:31.518 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.215 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:31.529 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.925 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:32.184 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.926 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:32.185 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.939 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:32.197 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.972 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:32.285 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.973 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:32.286 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:17.987 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:32.300 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:18.056 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:32.369 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:18.057 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:32.370 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:18.070 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:32.384 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:32.184 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:46.497 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:32.189 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:46.503 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:32.207 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:46.520 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:32.236 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:46.495 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:32.240 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:46.498 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:32.254 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:46.512 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:32.256 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:46.570 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:32.261 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:46.575 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:32.282 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:46.596 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:37.192 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:51.505 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:37.195 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:51.508 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:37.206 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:41:51.519 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:37.238 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:51.496 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:37.241 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:51.500 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:37.241 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:51.555 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:37.246 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:51.560 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:37.252 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:41:51.511 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:37.262 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:41:51.576 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:52.182 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:06.496 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:52.232 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:06.490 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:52.233 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:06.492 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:52.246 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:06.505 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:52.252 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:06.565 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:52.279 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:06.593 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:52.332 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:06.645 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:52.336 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:06.649 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:52.352 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:06.665 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:57.256 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:11.569 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:57.257 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:11.570 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:57.270 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:11.584 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:57.303 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:11.561 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:57.305 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:11.564 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:57.308 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:11.621 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:57.311 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:11.625 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:57.316 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:11.575 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:00:57.323 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:11.637 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:12.298 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:26.611 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:12.301 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:26.614 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:12.316 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:26.629 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:12.344 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:26.603 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:12.346 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:26.604 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:12.360 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:26.619 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:12.371 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:26.685 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:12.373 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:26.686 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:12.390 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:26.703 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:17.315 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:31.628 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:17.317 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:31.630 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:17.328 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:31.641 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:17.359 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:31.617 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:17.361 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:31.620 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:17.369 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:31.682 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:17.370 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:31.684 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:17.372 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:31.631 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:17.382 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:31.695 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:32.399 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:46.712 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:32.401 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:46.714 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:32.416 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:46.729 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:32.446 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:46.705 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:32.449 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:46.708 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:32.454 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:46.767 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:32.459 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:46.772 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:32.464 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:46.723 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:32.477 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:46.790 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:37.391 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:51.704 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:37.397 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:51.710 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:37.411 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:42:51.724 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:37.444 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:51.703 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:37.447 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:51.706 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:37.448 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:51.761 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:37.449 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:51.763 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:37.458 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:42:51.717 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:37.461 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:42:51.775 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:52.445 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:06.758 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:52.451 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:06.764 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:52.467 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:06.780 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:52.495 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:06.753 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:52.499 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:06.758 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:52.508 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:06.821 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:52.509 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:06.823 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:52.520 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:06.778 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:52.523 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:06.837 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:57.410 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:11.723 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:57.465 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:11.778 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:57.482 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:11.795 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:57.513 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:11.771 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:57.514 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:11.773 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:57.518 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:11.832 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:57.524 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:11.837 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:57.529 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:11.788 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:01:57.536 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:11.849 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:12.505 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:26.818 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:12.509 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:26.822 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:12.526 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:26.839 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:12.552 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:26.811 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:12.555 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:26.814 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:12.557 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:26.870 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:12.574 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:26.887 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:12.576 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:26.834 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:12.590 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:26.904 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:17.564 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:31.877 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:17.566 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:31.879 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:17.576 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:31.890 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:17.611 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:31.924 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:17.614 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:31.927 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:17.615 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:31.873 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:17.616 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:31.875 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:17.629 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:31.942 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:17.630 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:31.888 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:32.557 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:46.870 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:32.613 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:46.925 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:32.629 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:46.943 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:32.660 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:46.918 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:32.661 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:46.920 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:32.667 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:46.980 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:32.682 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:46.940 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:32.695 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:47.005 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:32.715 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:47.028 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:37.627 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:51.940 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:37.628 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:51.941 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:37.643 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:43:51.956 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:37.671 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:51.930 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:37.674 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:51.933 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:37.684 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:51.997 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:37.685 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:51.998 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:37.689 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:43:51.948 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:37.697 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:43:52.010 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:52.669 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:06.982 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:52.674 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:06.987 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:52.692 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:07.005 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:52.723 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:06.981 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:52.725 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:06.983 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:52.729 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:07.042 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:52.731 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:07.044 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:52.745 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:07.004 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:02:52.747 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:07.060 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:02.714 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:17.027 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:02.718 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:17.032 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:02.734 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:17.047 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:02.762 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:17.020 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:02.764 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:17.022 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:02.776 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:17.034 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:02.789 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:17.102 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:02.792 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:17.105 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:02.805 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:17.118 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:12.758 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:27.071 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:12.763 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:27.076 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:12.786 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:27.099 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:12.805 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:27.063 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:12.807 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:27.066 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:12.820 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:27.133 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:12.820 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:27.078 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:12.822 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:27.135 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:12.835 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:27.148 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:22.789 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:37.101 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:22.794 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:37.107 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:22.811 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:37.124 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:22.840 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:37.098 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:22.842 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:37.101 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:22.844 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:37.157 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:22.847 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:37.160 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:22.855 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:37.113 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:22.860 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:37.173 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:29.921 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:44.233 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:29.926 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:44.239 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:29.942 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:44.255 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:29.978 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:44.236 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:29.979 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:44.238 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:29.981 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:44.294 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:29.982 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:44.295 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:29.994 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:44.253 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:29.998 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:44.311 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:32.822 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:47.135 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:32.825 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:47.138 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:32.838 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:47.151 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:32.871 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:47.128 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:32.872 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:47.130 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:32.881 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:47.139 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:32.894 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:47.207 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:32.898 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:47.211 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:32.909 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:47.222 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:42.851 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:57.164 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:42.854 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:57.167 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:42.869 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:44:57.182 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:42.900 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:57.157 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:42.902 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:57.161 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:42.906 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:57.219 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:42.909 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:57.222 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:42.915 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:44:57.173 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:42.922 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:44:57.235 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:50.951 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-01-21T04:03:52.670 INFO:tasks.ceph.osd.4:Stopped 2024-01-21T04:03:52.671 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-21T04:03:52.671 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T04:03:52.685 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T04:03:52.673+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 34514) UID: 0 2024-01-21T04:03:52.688 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-01-21T04:03:52.675+0000 b47e640 -1 osd.5 1308 *** Got signal Terminated *** 2024-01-21T04:03:52.883 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:45:07.195 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:52.887 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:45:07.200 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:52.916 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:45:07.229 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:52.932 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:45:07.189 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:52.936 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:45:07.194 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:52.941 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:45:07.254 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:52.942 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:45:07.255 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:52.950 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:45:07.208 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:52.955 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:45:07.268 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:56.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-01-21T04:03:56.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-01-21T04:03:57.901 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:45:12.213 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:57.906 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:45:12.219 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:57.922 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:45:12.234 34172== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:57.957 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:45:12.270 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:57.959 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:45:12.271 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:57.963 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:45:12.219 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:57.964 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:45:12.221 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:57.972 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:45:12.285 34169== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:57.976 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:45:12.234 33978== Warning: unimplemented fcntl command: 1036 2024-01-21T04:03:58.772 INFO:tasks.ceph.osd.5:Stopped 2024-01-21T04:03:58.773 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-21T04:03:58.773 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T04:03:58.786 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T04:03:58.775+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 34517) UID: 0 2024-01-21T04:03:58.788 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-01-21T04:03:58.776+0000 b47e640 -1 osd.6 1310 *** Got signal Terminated *** 2024-01-21T04:04:02.263 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-01-21T04:04:02.263 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-01-21T04:04:04.875 INFO:tasks.ceph.osd.6:Stopped 2024-01-21T04:04:04.875 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-21T04:04:04.875 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T04:04:04.889 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-01-21T04:04:04.877+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 34520) UID: 0 2024-01-21T04:04:04.889 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-01-21T04:04:04.879+0000 b47e640 -1 osd.7 1310 *** Got signal Terminated *** 2024-01-21T04:04:07.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-01-21T04:04:07.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-01-21T04:04:10.976 INFO:tasks.ceph.osd.7:Stopped 2024-01-21T04:04:10.977 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-21T04:04:10.977 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-21T04:04:10.977 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T04:04:11.005 INFO:tasks.ceph.mgr.y:Stopped 2024-01-21T04:04:11.005 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-21T04:04:11.005 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T04:04:11.031 INFO:tasks.ceph.mgr.x:Stopped 2024-01-21T04:04:11.031 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-21T04:04:11.031 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-21T04:04:11.031 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T04:04:11.061 INFO:tasks.ceph.mon.a.smithi007.stderr:2024-01-21T04:04:11.047+0000 9f77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34140) UID: 0 2024-01-21T04:04:11.061 INFO:tasks.ceph.mon.a.smithi007.stderr:2024-01-21T04:04:11.049+0000 9f77640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T04:04:11.901 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:45:26.214 34172== 2024-01-21T04:04:11.901 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:45:26.214 34172== Exit program on first error (--exit-on-first-error=yes) 2024-01-21T04:04:11.925 INFO:tasks.ceph.mon.a.smithi007.stderr:daemon-helper: command failed with exit status 42 2024-01-21T04:04:12.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-01-21T04:04:12.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-01-21T04:04:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-01-21T04:04:17.134 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-01-21T04:04:17.135 INFO:tasks.ceph.mon.a:Stopped 2024-01-21T04:04:17.135 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-21T04:04:17.135 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T04:04:17.151 INFO:tasks.ceph.mon.c.smithi007.stderr:2024-01-21T04:04:17.138+0000 9f77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34142) UID: 0 2024-01-21T04:04:17.153 INFO:tasks.ceph.mon.c.smithi007.stderr:2024-01-21T04:04:17.139+0000 9f77640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-01-21T04:04:17.803 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:45:32.115 34169== 2024-01-21T04:04:17.803 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:45:32.115 34169== Exit program on first error (--exit-on-first-error=yes) 2024-01-21T04:04:17.825 INFO:tasks.ceph.mon.c.smithi007.stderr:daemon-helper: command failed with exit status 42 2024-01-21T04:04:18.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-01-21T04:04:18.073 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-01-21T04:04:23.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~5s 2024-01-21T04:04:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-01-21T04:04:23.242 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-01-21T04:04:23.243 INFO:tasks.ceph.mon.c:Stopped 2024-01-21T04:04:23.243 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-21T04:04:23.243 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T04:04:23.261 INFO:tasks.ceph.mon.b.smithi177.stderr:2024-01-21T04:04:23.245+0000 9f77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 33973) UID: 0 2024-01-21T04:04:23.261 INFO:tasks.ceph.mon.b.smithi177.stderr:2024-01-21T04:04:23.247+0000 9f77640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-01-21T04:04:23.976 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:45:38.234 33978== 2024-01-21T04:04:23.976 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:45:38.234 33978== Exit program on first error (--exit-on-first-error=yes) 2024-01-21T04:04:23.995 INFO:tasks.ceph.mon.b.smithi177.stderr:daemon-helper: command failed with exit status 42 2024-01-21T04:04:28.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-01-21T04:04:28.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-01-21T04:04:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-01-21T04:04:29.345 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi177 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-01-21T04:04:29.345 INFO:tasks.ceph.mon.b:Stopped 2024-01-21T04:04:29.346 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-21T04:04:29.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-01-21T04:04:29.383 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi007.front.sepia.ceph.com 2024-01-21T04:04:29.384 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-21T04:04:29.708 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi007.front.sepia.ceph.com 2024-01-21T04:04:29.708 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-21T04:04:29.803 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi007.front.sepia.ceph.com 2024-01-21T04:04:29.803 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-21T04:04:29.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi007.front.sepia.ceph.com 2024-01-21T04:04:29.868 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-21T04:04:29.927 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi177.front.sepia.ceph.com 2024-01-21T04:04:29.927 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-01-21T04:04:30.122 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi177.front.sepia.ceph.com 2024-01-21T04:04:30.123 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-01-21T04:04:30.187 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi177.front.sepia.ceph.com 2024-01-21T04:04:30.187 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-01-21T04:04:30.235 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi177.front.sepia.ceph.com 2024-01-21T04:04:30.235 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-01-21T04:04:30.285 INFO:tasks.ceph:Archiving mon data... 2024-01-21T04:04:30.286 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652/data/mon.a.tgz 2024-01-21T04:04:30.286 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-01-21T04:04:30.302 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.4fPfCvvdMq 2024-01-21T04:04:30.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.4fPfCvvdMq 2024-01-21T04:04:30.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.4fPfCvvdMq 2024-01-21T04:04:30.567 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.4fPfCvvdMq is 469KB 2024-01-21T04:04:30.635 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.4fPfCvvdMq 2024-01-21T04:04:30.650 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652/data/mon.c.tgz 2024-01-21T04:04:30.650 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-01-21T04:04:30.703 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.VBNY1s2iIy 2024-01-21T04:04:30.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.VBNY1s2iIy 2024-01-21T04:04:30.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.VBNY1s2iIy 2024-01-21T04:04:30.933 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.VBNY1s2iIy is 470KB 2024-01-21T04:04:31.031 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.VBNY1s2iIy 2024-01-21T04:04:31.045 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652/data/mon.b.tgz 2024-01-21T04:04:31.045 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-01-21T04:04:31.063 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.XgM9O0ikbC 2024-01-21T04:04:31.063 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.XgM9O0ikbC 2024-01-21T04:04:31.233 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.XgM9O0ikbC 2024-01-21T04:04:31.316 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.XgM9O0ikbC is 470KB 2024-01-21T04:04:31.412 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.XgM9O0ikbC 2024-01-21T04:04:31.427 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-21T04:04:31.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-21T04:04:31.429 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-21T04:04:31.813 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-21T04:04:32.071 INFO:tasks.ceph:Archiving crash dumps... 2024-01-21T04:04:32.073 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652/remote/smithi007/crash 2024-01-21T04:04:32.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-21T04:04:32.101 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652/remote/smithi177/crash 2024-01-21T04:04:32.102 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-21T04:04:32.128 INFO:tasks.ceph:Compressing logs... 2024-01-21T04:04:32.128 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T04:04:32.145 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T04:04:32.169 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38676.log 2024-01-21T04:04:32.170 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39253.log 2024-01-21T04:04:32.170 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38676.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-21T04:04:32.174 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81207.log 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mon.a.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.38676.log.gz 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.107706.log 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39253.log.gz 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103178.log 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.81207.log: /var/log/ceph/ceph-client.admin.107706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41022.log 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.103178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38911.log 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107706.log.gz 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40197.log 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39213.log 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41397.log 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38911.log.gz/var/log/ceph/ceph-client.admin.39213.log: 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40245.log 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34484.log 2024-01-21T04:04:32.175 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5%/var/log/ceph/ceph-client.admin.41397.log: -- replaced with /var/log/ceph/ceph-client.admin.39213.log.gz 2024-01-21T04:04:32.176 INFO:teuthology.orchestra.run.smithi007.stderr: 90.6%/var/log/ceph/ceph-client.admin.40245.log: -- replaced with /var/log/ceph/ceph-client.admin.81207.log.gz 2024-01-21T04:04:32.176 INFO:teuthology.orchestra.run.smithi007.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.103178.log.gz 2024-01-21T04:04:32.176 INFO:teuthology.orchestra.run.smithi007.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.40245.log.gz 2024-01-21T04:04:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107832.log 2024-01-21T04:04:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40168.log 2024-01-21T04:04:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.107832.log: 87.4%gzip 91.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.41022.log.gz --verbose 2024-01-21T04:04:32.187 INFO:teuthology.orchestra.run.smithi007.stderr: -- /var/log/ceph/ceph-client.admin.38332.log 2024-01-21T04:04:32.187 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34484.log.gz 2024-01-21T04:04:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40168.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-01-21T04:04:32.187 INFO:teuthology.orchestra.run.smithi007.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.107832.log.gz 2024-01-21T04:04:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:gzip/var/log/ceph/ceph-client.admin.38332.log: -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-01-21T04:04:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-01-21T04:04:32.187 INFO:teuthology.orchestra.run.smithi007.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-01-21T04:04:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/osd.2.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38332.log.gz 2024-01-21T04:04:32.187 INFO:teuthology.orchestra.run.smithi007.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-01-21T04:04:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-01-21T04:04:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/osd.0.log: /var/log/ceph/valgrind/osd.1.log: gzip -5 63.9% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr: --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/mon.a.log: 69.5%gzip -- replaced with /var/log/ceph/valgrind/mon.a.log.gz -5 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr: --verbose -- /var/log/ceph/ceph-client.admin.81209.log 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose --/var/log/ceph/valgrind/osd.3.log: /var/log/ceph/ceph-client.admin.38115.log 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.81209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40858.log 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108378.log 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40858.log: 93.1% -- replaced with /var/log/ceph/ceph-client.admin.41397.log.gz 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38115.log.gz 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39628.log 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.108378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38946.log 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39628.log: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108420.log 2024-01-21T04:04:32.188 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.108378.log.gz 2024-01-21T04:04:32.189 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5%/var/log/ceph/ceph-client.admin.38946.log: -- replaced with /var/log/ceph/ceph-client.admin.39628.log.gz 2024-01-21T04:04:32.189 INFO:teuthology.orchestra.run.smithi007.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.81209.log.gz 2024-01-21T04:04:32.189 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38946.log.gz 2024-01-21T04:04:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40385.log 2024-01-21T04:04:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.108420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39843.log 2024-01-21T04:04:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67246.log 2024-01-21T04:04:32.189 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108420.log.gz 2024-01-21T04:04:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41049.log 2024-01-21T04:04:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.67246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41238.log 2024-01-21T04:04:32.189 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39843.log.gz 2024-01-21T04:04:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41049.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.67246.log.gz 2024-01-21T04:04:32.189 INFO:teuthology.orchestra.run.smithi007.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41049.log.gz 2024-01-21T04:04:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38372.log 2024-01-21T04:04:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108084.log 2024-01-21T04:04:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41238.log: /var/log/ceph/ceph-client.admin.38372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107538.log 2024-01-21T04:04:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.108084.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38372.log.gz 2024-01-21T04:04:32.191 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108084.log.gz 2024-01-21T04:04:32.195 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34423.log 2024-01-21T04:04:32.195 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-01-21T04:04:32.206 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37888.log 2024-01-21T04:04:32.206 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.107538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41028.log 2024-01-21T04:04:32.206 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37888.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.107538.log.gz 2024-01-21T04:04:32.206 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37888.log.gz 2024-01-21T04:04:32.207 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34423.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-01-21T04:04:32.207 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-01-21T04:04:32.207 INFO:teuthology.orchestra.run.smithi177.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34423.log.gz 2024-01-21T04:04:32.207 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-01-21T04:04:32.207 INFO:teuthology.orchestra.run.smithi177.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-01-21T04:04:32.207 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-01-21T04:04:32.207 INFO:teuthology.orchestra.run.smithi177.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-01-21T04:04:32.207 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-01-21T04:04:32.207 INFO:teuthology.orchestra.run.smithi177.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-01-21T04:04:32.207 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34252.log 2024-01-21T04:04:32.207 INFO:teuthology.orchestra.run.smithi177.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-01-21T04:04:32.207 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34212.log 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34171.log 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34212.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34252.log.gz 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log/var/log/ceph/ceph-client.admin.34171.log: 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr: 88.0%/var/log/ceph/ceph.audit.log: -- replaced with /var/log/ceph/ceph-client.admin.34212.log.gz 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34171.log.gz 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34372.log 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34131.log 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34372.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr: 88.7%/var/log/ceph/ceph-client.admin.34131.log: -- replaced with /var/log/ceph/ceph-client.admin.34372.log.gz 2024-01-21T04:04:32.208 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34332.log 2024-01-21T04:04:32.209 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mgr.x.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34131.log.gz 2024-01-21T04:04:32.209 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34292.log 2024-01-21T04:04:32.209 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34332.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34332.log.gz 2024-01-21T04:04:32.209 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-21T04:04:32.209 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34292.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34292.log.gz 2024-01-21T04:04:32.209 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-01-21T04:04:32.210 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.log: 92.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-21T04:04:32.211 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108460.log 2024-01-21T04:04:32.211 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40724.log 2024-01-21T04:04:32.213 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.108460.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108460.log.gz 2024-01-21T04:04:32.213 INFO:teuthology.orchestra.run.smithi007.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.41028.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40771.log 2024-01-21T04:04:32.213 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T04:04:32.213 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40724.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.40168.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38668.log 2024-01-21T04:04:32.213 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T04:04:32.213 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94177.log 2024-01-21T04:04:32.214 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38668.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38668.log.gz 2024-01-21T04:04:32.214 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39549.log 2024-01-21T04:04:32.215 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-21T04:04:32.218 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94177.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.94177.log.gz 2024-01-21T04:04:32.218 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67763.log 2024-01-21T04:04:32.219 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39549.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39549.log.gz 2024-01-21T04:04:32.219 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41379.log 2024-01-21T04:04:32.219 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.67763.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40724.log.gz 2024-01-21T04:04:32.222 INFO:teuthology.orchestra.run.smithi007.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40771.log.gz 2024-01-21T04:04:32.222 INFO:teuthology.orchestra.run.smithi007.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.67763.log.gz 2024-01-21T04:04:32.223 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94172.log 2024-01-21T04:04:32.223 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39709.log 2024-01-21T04:04:32.223 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41207.log 2024-01-21T04:04:32.224 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39709.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39709.log.gz 2024-01-21T04:04:32.225 INFO:teuthology.orchestra.run.smithi007.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.41238.log.gz 2024-01-21T04:04:32.226 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39729.log 2024-01-21T04:04:32.226 INFO:teuthology.orchestra.run.smithi007.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.94172.log.gz 2024-01-21T04:04:32.226 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-21T04:04:32.227 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41207.log: /var/log/ceph/ceph-client.admin.39729.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39729.log.gz 2024-01-21T04:04:32.227 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41207.log.gz 2024-01-21T04:04:32.228 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40460.log 2024-01-21T04:04:32.228 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41287.log 2024-01-21T04:04:32.228 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.audit.log: gzip/var/log/ceph/ceph-client.admin.40460.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39629.log 2024-01-21T04:04:32.232 INFO:teuthology.orchestra.run.smithi177.stderr: 94.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-21T04:04:32.234 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41287.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.41379.log.gz 2024-01-21T04:04:32.234 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40360.log 2024-01-21T04:04:32.237 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39629.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39629.log.gz 2024-01-21T04:04:32.237 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69672.log 2024-01-21T04:04:32.237 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40360.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40858.log.gz 2024-01-21T04:04:32.237 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28331.log 2024-01-21T04:04:32.240 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.69672.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.69672.log.gz 2024-01-21T04:04:32.240 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40363.log 2024-01-21T04:04:32.241 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.tmp-client.admin.28331.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28331.log.gz 2024-01-21T04:04:32.241 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38273.log 2024-01-21T04:04:32.245 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40363.log: 95.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-21T04:04:32.246 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41140.log 2024-01-21T04:04:32.246 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38273.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38273.log.gz 2024-01-21T04:04:32.248 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39293.log 2024-01-21T04:04:32.248 INFO:teuthology.orchestra.run.smithi007.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.40363.log.gz 2024-01-21T04:04:32.248 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103177.log 2024-01-21T04:04:32.249 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39293.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39293.log.gz 2024-01-21T04:04:32.249 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81208.log 2024-01-21T04:04:32.250 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.103177.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40197.log.gz 2024-01-21T04:04:32.251 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39589.log 2024-01-21T04:04:32.252 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.81208.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.103177.log.gz 2024-01-21T04:04:32.253 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107874.log 2024-01-21T04:04:32.253 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39589.log: 87.4% 91.2% -- replaced with /var/log/ceph/ceph-client.admin.81208.log.gz 2024-01-21T04:04:32.253 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39589.log.gz 2024-01-21T04:04:32.254 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40272.log 2024-01-21T04:04:32.254 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.107874.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-21T04:04:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40272.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107874.log.gz 2024-01-21T04:04:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41023.log 2024-01-21T04:04:32.258 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.1.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.41287.log.gz 2024-01-21T04:04:32.267 INFO:teuthology.orchestra.run.smithi007.stderr: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.41140.log.gz 2024-01-21T04:04:32.268 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 2024-01-21T04:04:32.268 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107664.log 2024-01-21T04:04:32.272 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41023.log: /var/log/ceph/ceph-client.admin.40720.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41023.log.gz 2024-01-21T04:04:32.272 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103175.log 2024-01-21T04:04:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.107664.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107664.log.gz 2024-01-21T04:04:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94176.log 2024-01-21T04:04:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.103175.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2024-01-21T04:04:32.277 INFO:teuthology.orchestra.run.smithi007.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.103175.log.gz 2024-01-21T04:04:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41391.log 2024-01-21T04:04:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38680.log 2024-01-21T04:04:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94176.log: /var/log/ceph/ceph-client.admin.41391.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.94176.log.gz 2024-01-21T04:04:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108126.log 2024-01-21T04:04:32.282 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38680.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38680.log.gz 2024-01-21T04:04:32.283 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34586.log 2024-01-21T04:04:32.283 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.108126.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108126.log.gz 2024-01-21T04:04:32.284 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35501.log 2024-01-21T04:04:32.284 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34586.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34586.log.gz 2024-01-21T04:04:32.285 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39885.log 2024-01-21T04:04:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35501.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35501.log.gz 2024-01-21T04:04:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40238.log 2024-01-21T04:04:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39885.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39885.log.gz 2024-01-21T04:04:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40644.log 2024-01-21T04:04:32.297 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40238.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.40238.log.gz 2024-01-21T04:04:32.298 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41317.log 2024-01-21T04:04:32.302 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40644.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40644.log.gz 2024-01-21T04:04:32.303 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38415.log 2024-01-21T04:04:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108210.log 2024-01-21T04:04:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38415.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38415.log.gz 2024-01-21T04:04:32.318 INFO:teuthology.orchestra.run.smithi007.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.40272.log.gz 2024-01-21T04:04:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-01-21T04:04:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.108210.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108210.log.gz 2024-01-21T04:04:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38457.log 2024-01-21T04:04:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.3.log: 94.4% -- replaced with /var/log/ceph/ceph-client.admin.40460.log.gz 2024-01-21T04:04:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38906.log 2024-01-21T04:04:32.328 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38457.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38457.log.gz 2024-01-21T04:04:32.328 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107622.log 2024-01-21T04:04:32.329 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38906.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38906.log.gz 2024-01-21T04:04:32.329 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38670.log 2024-01-21T04:04:32.330 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.107622.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107622.log.gz 2024-01-21T04:04:32.330 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39423.log 2024-01-21T04:04:32.330 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38670.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.41391.log.gz 2024-01-21T04:04:32.331 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38670.log.gz 2024-01-21T04:04:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38499.log 2024-01-21T04:04:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39801.log 2024-01-21T04:04:32.333 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38499.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39423.log.gz 2024-01-21T04:04:32.333 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38499.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38681.log 2024-01-21T04:04:32.333 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T04:04:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39801.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39801.log.gz 2024-01-21T04:04:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108294.log 2024-01-21T04:04:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38216.log 2024-01-21T04:04:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.108294.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38681.log.gz 2024-01-21T04:04:32.335 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108294.log.gz 2024-01-21T04:04:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38678.log 2024-01-21T04:04:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38216.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38216.log.gz 2024-01-21T04:04:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67764.log 2024-01-21T04:04:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40276.log 2024-01-21T04:04:32.337 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38678.log: /var/log/ceph/ceph-client.admin.67764.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38678.log.gz 2024-01-21T04:04:32.339 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39376.log 2024-01-21T04:04:32.339 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67764.log.gz 2024-01-21T04:04:32.339 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34705.log 2024-01-21T04:04:32.339 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39376.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39376.log.gz 2024-01-21T04:04:32.339 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39393.log 2024-01-21T04:04:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34705.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34705.log.gz 2024-01-21T04:04:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40556.log 2024-01-21T04:04:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39393.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39393.log.gz 2024-01-21T04:04:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40443.log 2024-01-21T04:04:32.343 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40556.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.40276.log.gz 2024-01-21T04:04:32.344 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39054.log 2024-01-21T04:04:32.350 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40443.log: 96.6% -- replaced with /var/log/ceph/ceph-client.admin.41317.log.gz 2024-01-21T04:04:32.352 INFO:teuthology.orchestra.run.smithi007.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.40556.log.gz 2024-01-21T04:04:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40964.log 2024-01-21T04:04:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39054.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-01-21T04:04:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40964.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40443.log.gz 2024-01-21T04:04:32.353 INFO:teuthology.orchestra.run.smithi007.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39054.log.gz 2024-01-21T04:04:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36540.log 2024-01-21T04:04:32.353 INFO:teuthology.orchestra.run.smithi007.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40964.log.gz 2024-01-21T04:04:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108252.log 2024-01-21T04:04:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.36540.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36540.log.gz 2024-01-21T04:04:32.356 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40810.log 2024-01-21T04:04:32.356 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.108252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38893.log 2024-01-21T04:04:32.356 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40810.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108252.log.gz 2024-01-21T04:04:32.356 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107916.log 2024-01-21T04:04:32.356 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38893.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38893.log.gz 2024-01-21T04:04:32.356 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41026.log 2024-01-21T04:04:32.358 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.107916.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107916.log.gz 2024-01-21T04:04:32.358 INFO:teuthology.orchestra.run.smithi007.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.40810.log.gz 2024-01-21T04:04:32.358 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107958.log 2024-01-21T04:04:32.358 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41200.log 2024-01-21T04:04:32.359 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41026.log: /var/log/ceph/ceph-client.admin.107958.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107958.log.gz 2024-01-21T04:04:32.359 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40634.log 2024-01-21T04:04:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41200.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.41026.log.gz 2024-01-21T04:04:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108168.log 2024-01-21T04:04:32.373 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40634.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.41200.log.gz 2024-01-21T04:04:32.373 INFO:teuthology.orchestra.run.smithi007.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40634.log.gz 2024-01-21T04:04:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-21T04:04:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.108168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94182.log 2024-01-21T04:04:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.0.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108168.log.gz 2024-01-21T04:04:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34182.log 2024-01-21T04:04:32.377 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94182.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.94182.log.gz 2024-01-21T04:04:32.377 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67248.log 2024-01-21T04:04:32.378 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34182.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.34182.log.gz 2024-01-21T04:04:32.378 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108336.log 2024-01-21T04:04:32.381 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.67248.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.67248.log.gz 2024-01-21T04:04:32.381 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108042.log 2024-01-21T04:04:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.108336.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108336.log.gz 2024-01-21T04:04:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67247.log 2024-01-21T04:04:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.108042.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108042.log.gz 2024-01-21T04:04:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40209.log 2024-01-21T04:04:32.386 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.67247.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.67247.log.gz 2024-01-21T04:04:32.386 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39474.log 2024-01-21T04:04:32.395 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40209.log: 91.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40502.log 2024-01-21T04:04:32.395 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39474.log: -- replaced with /var/log/ceph/ceph-client.admin.40209.log.gz 2024-01-21T04:04:32.397 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39474.log.gz 2024-01-21T04:04:32.397 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-21T04:04:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40502.log: 91.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41323.log 2024-01-21T04:04:32.422 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39049.log 2024-01-21T04:04:32.422 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40502.log.gz 2024-01-21T04:04:32.427 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41323.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-21T04:04:32.427 INFO:teuthology.orchestra.run.smithi007.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41323.log.gz 2024-01-21T04:04:32.427 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41059.log 2024-01-21T04:04:32.429 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39049.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39049.log.gz 2024-01-21T04:04:32.429 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-21T04:04:32.437 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41059.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41059.log.gz 2024-01-21T04:04:32.437 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107790.log 2024-01-21T04:04:32.451 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39313.log 2024-01-21T04:04:32.451 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.107790.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107790.log.gz 2024-01-21T04:04:32.465 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40543.log 2024-01-21T04:04:32.465 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39313.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39313.log.gz 2024-01-21T04:04:32.478 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40848.log 2024-01-21T04:04:32.484 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40543.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40543.log.gz 2024-01-21T04:04:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38679.log 2024-01-21T04:04:32.498 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40848.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40848.log.gz 2024-01-21T04:04:32.508 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107748.log 2024-01-21T04:04:32.509 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38679.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38679.log.gz 2024-01-21T04:04:32.515 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41346.log 2024-01-21T04:04:32.516 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.107748.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107748.log.gz 2024-01-21T04:04:32.524 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40980.log 2024-01-21T04:04:32.542 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40728.log 2024-01-21T04:04:32.542 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40980.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40980.log.gz 2024-01-21T04:04:32.550 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38675.log 2024-01-21T04:04:32.559 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40728.log: 90.7% 91.6% -- replaced with /var/log/ceph/ceph-client.admin.41346.log.gz 2024-01-21T04:04:32.563 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39125.log 2024-01-21T04:04:32.564 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38675.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38675.log.gz 2024-01-21T04:04:32.571 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40728.log.gz 2024-01-21T04:04:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107580.log 2024-01-21T04:04:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39125.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39125.log.gz 2024-01-21T04:04:32.589 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108000.log 2024-01-21T04:04:32.593 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.107580.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.107580.log.gz 2024-01-21T04:04:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40903.log 2024-01-21T04:04:32.606 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.108000.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108000.log.gz 2024-01-21T04:04:32.614 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69674.log 2024-01-21T04:04:32.626 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38951.log 2024-01-21T04:04:32.635 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.69674.log: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94175.log 2024-01-21T04:04:32.635 INFO:teuthology.orchestra.run.smithi007.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.40903.log.gz 2024-01-21T04:04:32.641 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38951.log: -- replaced with /var/log/ceph/ceph-client.admin.69674.log.gz 2024-01-21T04:04:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40979.log 2024-01-21T04:04:32.647 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.94175.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38951.log.gz 2024-01-21T04:04:32.658 INFO:teuthology.orchestra.run.smithi007.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.94175.log.gz 2024-01-21T04:04:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-01-21T04:04:32.665 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40979.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.40979.log.gz 2024-01-21T04:04:32.669 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69673.log 2024-01-21T04:04:32.692 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.69673.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.69673.log.gz 2024-01-21T04:04:32.782 INFO:teuthology.orchestra.run.smithi007.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.40360.log.gz 2024-01-21T04:04:32.864 INFO:teuthology.orchestra.run.smithi007.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-01-21T04:04:33.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-01-21T04:04:33.342 INFO:teuthology.orchestra.run.smithi007.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40385.log.gz 2024-01-21T04:04:33.882 INFO:teuthology.orchestra.run.smithi177.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-01-21T04:04:34.827 INFO:teuthology.orchestra.run.smithi007.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-01-21T04:04:35.960 INFO:teuthology.orchestra.run.smithi007.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-21T04:04:38.381 INFO:teuthology.orchestra.run.smithi177.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-01-21T04:04:38.391 INFO:teuthology.orchestra.run.smithi177.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-01-21T04:04:38.934 INFO:teuthology.orchestra.run.smithi007.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-21T04:04:39.114 INFO:teuthology.orchestra.run.smithi177.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-01-21T04:04:39.170 INFO:teuthology.orchestra.run.smithi007.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-21T04:04:39.464 INFO:teuthology.orchestra.run.smithi177.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-01-21T04:04:39.466 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-01-21T04:04:39.466 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m7.283s 2024-01-21T04:04:39.466 INFO:teuthology.orchestra.run.smithi177.stderr:user 0m27.351s 2024-01-21T04:04:39.466 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m0.919s 2024-01-21T04:04:39.472 INFO:teuthology.orchestra.run.smithi007.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-21T04:04:40.721 INFO:teuthology.orchestra.run.smithi007.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-01-21T04:04:40.723 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-01-21T04:04:40.723 INFO:teuthology.orchestra.run.smithi007.stderr:real 0m8.566s 2024-01-21T04:04:40.723 INFO:teuthology.orchestra.run.smithi007.stderr:user 0m36.579s 2024-01-21T04:04:40.723 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m1.187s 2024-01-21T04:04:40.723 INFO:tasks.ceph:Archiving logs... 2024-01-21T04:04:40.723 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652/remote/smithi007/log 2024-01-21T04:04:40.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T04:04:45.601 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652/remote/smithi177/log 2024-01-21T04:04:45.602 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T04:04:49.668 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-21T04:04:49.669 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-21T04:04:49.678 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-21T04:04:49.678 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-21T04:04:49.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T04:04:49.682 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T04:04:49.760 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-21T04:04:49.768 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-21T04:04:49.768 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T04:04:49.771 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T04:04:49.785 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-01-21T04:04:49.787 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-01-21T04:04:49.789 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T04:04:49.790 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-01-21T04:04:49.790 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 -145us[ -145us] +/- 82ms 2024-01-21T04:04:49.790 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -554us[ -570us] +/- 52ms 2024-01-21T04:04:49.790 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +816us[ +800us] +/- 65ms 2024-01-21T04:04:49.790 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T04:04:49.791 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T04:04:49.791 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-01-21T04:04:49.792 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 -185us[ -228us] +/- 82ms 2024-01-21T04:04:49.792 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -595us[ -638us] +/- 52ms 2024-01-21T04:04:49.792 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +773us[ +730us] +/- 65ms 2024-01-21T04:04:49.792 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T04:04:49.793 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-21T04:04:49.801 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-21T04:04:49.801 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-21T04:04:49.810 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T04:04:49.904 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T04:04:49.973 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T04:04:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T04:04:50.006 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T04:04:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T04:04:50.037 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-21T04:04:50.048 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-21T04:04:50.059 INFO:teuthology.task.internal:Duration was 3710.101512 seconds 2024-01-21T04:04:50.060 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-21T04:04:50.069 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-21T04:04:50.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T04:04:50.072 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T04:04:50.116 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T04:04:50.122 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T04:04:50.536 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-21T04:04:50.537 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-01-21T04:04:50.537 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T04:04:50.558 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-01-21T04:04:50.559 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T04:04:50.585 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-21T04:04:50.585 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T04:04:50.601 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T04:04:50.655 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-21T04:04:50.655 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T04:04:50.671 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T04:04:50.794 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-21T04:04:50.803 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-21T04:04:50.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T04:04:50.830 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T04:04:50.857 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-21T04:04:50.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T04:04:50.872 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T04:04:50.895 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-01-21T04:04:50.923 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-01-21T04:04:50.944 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T04:04:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T04:04:50.968 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T04:04:51.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T04:04:51.000 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-21T04:04:51.010 INFO:teuthology.task.internal:Transferring archived files... 2024-01-21T04:04:51.010 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652/remote/smithi007 2024-01-21T04:04:51.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T04:04:51.067 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652/remote/smithi177 2024-01-21T04:04:51.067 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T04:04:51.102 INFO:teuthology.task.internal:Removing archive directory... 2024-01-21T04:04:51.102 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T04:04:51.105 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T04:04:51.150 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-21T04:04:51.164 INFO:teuthology.task.internal:Not uploading archives. 2024-01-21T04:04:51.164 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-21T04:04:51.173 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-21T04:04:51.173 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T04:04:51.176 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T04:04:51.189 INFO:teuthology.orchestra.run.smithi007.stdout: 265814 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 04:04 /home/ubuntu/cephtest 2024-01-21T04:04:51.205 INFO:teuthology.orchestra.run.smithi177.stdout: 265291 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 04:04 /home/ubuntu/cephtest 2024-01-21T04:04:51.206 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-21T04:04:51.216 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-21T04:04:51.226 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-21T04:04:51.226 INFO:teuthology.nuke:Checking targets against current locks 2024-01-21T04:04:51.252 DEBUG:teuthology.nuke:shortname: smithi007 2024-01-21T04:04:51.252 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T04:04:51.275 DEBUG:teuthology.nuke:shortname: smithi177 2024-01-21T04:04:51.275 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T04:04:51.278 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 02:54:48.390935', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCysGnDiMG3hGIFgiXaTH/oCK9ndRzJ55SkMZMS2Q83b6pe6HQhs8gb6USSF4yFdUyIfEBXiIdb/fyyj27NdoiU='} 2024-01-21T04:04:51.298 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_16:13:55-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523652', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 02:54:48.392521', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCnBrRXFdjEgSX7q1KhDE3anvh06bdhYQXczN5WiULNIIELABIokQnA9prKtCgCRS7uekcklFQ8pNArUFnIQJHc='} 2024-01-21T04:04:51.328 INFO:teuthology.orchestra.console.smithi007:Power off 2024-01-21T04:04:51.328 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T04:04:51.356 INFO:teuthology.orchestra.console.smithi177:Power off 2024-01-21T04:04:51.357 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T04:04:51.366 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2024-01-21T04:04:51.366 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T04:04:51.380 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-01-21T04:04:51.381 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T04:04:51.397 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-01-21T04:04:51.415 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-01-21T04:04:55.398 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T04:04:55.415 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T04:04:55.508 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-01-21T04:04:55.524 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-01-21T04:04:59.510 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T04:04:59.524 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T04:04:59.619 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-01-21T04:04:59.634 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-01-21T04:04:59.735 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-01-21T04:05:03.621 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T04:05:03.730 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-01-21T04:05:03.831 INFO:teuthology.orchestra.console.smithi007:Power off completed 2024-01-21T04:05:03.935 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} duration: 3710.1015117168427 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-01-21T04:05:03.936 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T04:05:04.013 INFO:teuthology.run:FAIL